[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:04 dist-test-slave-52fr krb5kdc[30037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:04 dist-test-slave-52fr krb5kdc[30037](info): set up 2 sockets
Sep 24 20:13:04 dist-test-slave-52fr krb5kdc[30037](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 20:13:05.681664 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:05 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:05.813194 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46827
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:05.930403 30050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:05.930764 30050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:05.930850 30050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:05.935122 30050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:05.935227 30050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:05.935266 30050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:05.935335 30050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:05.935374 30050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:05.940979 30050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:05.942274 30050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:05.943378 30050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:05.952621 30056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:05.952621 30058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:05.952615 30055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:05.952904 30050 server_base.cc:1034] running on GCE node
I20260924 20:13:05.953560 30050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:05.955004 30050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:05.958308 30050 hybrid_clock.cc:648] HybridClock initialized: now 1790280785958268 us; error 47 us; skew 500 ppm
Sep 24 20:13:05 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:05.962126 30050 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:05.963383 30050 webserver.cc:460] Webserver started at http://127.29.83.190:36159/ using document root <none> and password file <none>
I20260924 20:13:05.964123 30050 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:05.964216 30050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:05.964470 30050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:05.967550 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "d57e8f327bae438c97de2091f4b3ab18"
format_stamp: "Formatted at 2026-09-24 20:13:05 on dist-test-slave-52fr"
server_key: "a165e15de667eb52adc633a3d9f08817"
server_key_iv: "de877e84f3672379a9daaf3006b1f7d7"
server_key_version: "encryptionkey@0"
I20260924 20:13:05.968359 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "d57e8f327bae438c97de2091f4b3ab18"
format_stamp: "Formatted at 2026-09-24 20:13:05 on dist-test-slave-52fr"
server_key: "a165e15de667eb52adc633a3d9f08817"
server_key_iv: "de877e84f3672379a9daaf3006b1f7d7"
server_key_version: "encryptionkey@0"
I20260924 20:13:05.973886 30050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 20:13:05.977393 30065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:05.979393 30050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:05.979553 30050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "d57e8f327bae438c97de2091f4b3ab18"
format_stamp: "Formatted at 2026-09-24 20:13:05 on dist-test-slave-52fr"
server_key: "a165e15de667eb52adc633a3d9f08817"
server_key_iv: "de877e84f3672379a9daaf3006b1f7d7"
server_key_version: "encryptionkey@0"
I20260924 20:13:05.979708 30050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:06.003022 30050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:06.006947 30050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:06.007126 30050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:06.015110 30050 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:46827
I20260924 20:13:06.015213 30117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:46827 every 8 connection(s)
I20260924 20:13:06.016132 30050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:06.019842 30118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:06.020844 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30050
I20260924 20:13:06.021041 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:06.022464 30030 external_mini_cluster.cc:1428] Setting key 8b4fcb77cc4dc17887ec1989f3daa23d
I20260924 20:13:06.028033 30118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: Bootstrap starting.
I20260924 20:13:06.031029 30118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:06.031801 30118 log.cc:824] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:06.034094 30118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: No bootstrap required, opened a new log
I20260924 20:13:06.038214 30118 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } }
I20260924 20:13:06.038504 30118 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:06.038564 30118 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57e8f327bae438c97de2091f4b3ab18, State: Initialized, Role: FOLLOWER
I20260924 20:13:06.039067 30118 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } }
I20260924 20:13:06.039237 30118 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:06.039306 30118 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:06.039414 30118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:06.040411 30118 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } }
I20260924 20:13:06.040822 30118 leader_election.cc:302] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57e8f327bae438c97de2091f4b3ab18; no voters:
I20260924 20:13:06.041211 30118 leader_election.cc:288] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:06.041447 30123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:06.041808 30123 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [term 1 LEADER]: Becoming Leader. State: Replica: d57e8f327bae438c97de2091f4b3ab18, State: Running, Role: LEADER
I20260924 20:13:06.042232 30123 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } }
I20260924 20:13:06.042757 30118 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:06.044299 30125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57e8f327bae438c97de2091f4b3ab18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } } }
I20260924 20:13:06.044449 30125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:06.044940 30124 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57e8f327bae438c97de2091f4b3ab18. Latest consensus state: current_term: 1 leader_uuid: "d57e8f327bae438c97de2091f4b3ab18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57e8f327bae438c97de2091f4b3ab18" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46827 } } }
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:06.045092 30124 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18 [sys.catalog]: This master's current role is: LEADER
W20260924 20:13:06.048491 30138 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:06.048585 30138 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:06.048688 30133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:06.049599 30133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:06.055313 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.025821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.29.83.190:46827)
0924 20:13:06.026330 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.026351 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.027461 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 20:13:06.032061 (+ 4600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.032086 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.032551 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.033002 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.035983 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.037666 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.039467 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.040275 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.040479 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.046373 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.046383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.046404 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.046443 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.048965 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.049832 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.049840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.049849 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.050014 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.050506 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.050511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.050514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.051119 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.051362 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.053245 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.053566 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":156,"threads_started":1}
I20260924 20:13:06.058794 30133 catalog_manager.cc:1347] Generated new cluster ID: eb79cb37dd0449cebbe47ca22d352a5b
I20260924 20:13:06.058866 30133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:06.071206 30133 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:06.072530 30133 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:06.079586 30133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57e8f327bae438c97de2091f4b3ab18: Generated new TSK 0
I20260924 20:13:06.080703 30133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:06.117980 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:06.229396 30144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:06.229699 30144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:06.229745 30144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:06.233992 30144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:06.234053 30144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:06.234117 30144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:06.239403 30144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:06.240453 30144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:06.241441 30144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:06.251592 30150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.251573 30149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.251565 30152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:06.251698 30144 server_base.cc:1034] running on GCE node
I20260924 20:13:06.252178 30144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:06.252827 30144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:06.254036 30144 hybrid_clock.cc:648] HybridClock initialized: now 1790280786254001 us; error 47 us; skew 500 ppm
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:06.256650 30144 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:06.257489 30144 webserver.cc:460] Webserver started at http://127.29.83.129:36789/ using document root <none> and password file <none>
I20260924 20:13:06.258104 30144 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:06.258189 30144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:06.258417 30144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:06.260584 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "6ccdf27e41e342b6aad9c256c2c8eacd"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "d00c9342b7b45d051f98eae968a6d9a3"
server_key_iv: "684400954795b564cbc75b1b229e0333"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.261139 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "6ccdf27e41e342b6aad9c256c2c8eacd"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "d00c9342b7b45d051f98eae968a6d9a3"
server_key_iv: "684400954795b564cbc75b1b229e0333"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.265204 30144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:13:06.267678 30159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.268931 30144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:06.269064 30144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "6ccdf27e41e342b6aad9c256c2c8eacd"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "d00c9342b7b45d051f98eae968a6d9a3"
server_key_iv: "684400954795b564cbc75b1b229e0333"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.269184 30144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:06.289013 30144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:06.292415 30144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:06.292629 30144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:06.293252 30144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:06.294159 30144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:06.294229 30144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.294298 30144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:06.294346 30144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.303552 30144 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:36353
I20260924 20:13:06.303576 30272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:36353 every 8 connection(s)
I20260924 20:13:06.304473 30144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:06.314631 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30144
I20260924 20:13:06.314714 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:06.314973 30030 external_mini_cluster.cc:1428] Setting key fa26b9689d9e772f35b2c0c3428cf389
I20260924 20:13:06.319047 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.306370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34145 (local address 127.29.83.190:46827)
0924 20:13:06.306507 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.306510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.307350 (+ 840us) server_negotiation.cc:407] Connection header received
0924 20:13:06.308703 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.308707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.308769 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.308906 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.310230 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.310777 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.311946 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.312185 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.312299 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.314849 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.314852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.314855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.314880 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.316528 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.317161 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.317183 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.317185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.317284 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.317689 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.317691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.317692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.317920 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.317999 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.318669 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.318833 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:13:06.319906 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.306704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46827 (local address 127.29.83.129:34145)
0924 20:13:06.307171 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:06.307215 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:06.308470 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:06.309091 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:06.309103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:06.309499 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:06.310066 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.310080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.310919 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.310923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.311810 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.311820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.312363 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.312368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.312538 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.313111 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:06.313129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:06.314653 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:06.316695 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.316701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.316717 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.317066 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.317386 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.317392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.317396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.317592 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.318025 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:06.318035 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:06.318374 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 20:13:06.318607 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 20:13:06.318934 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260924 20:13:06.321337 30273 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46827
I20260924 20:13:06.321592 30273 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:06.322144 30273 heartbeater.cc:511] Master 127.29.83.190:46827 requested a full tablet report, sending...
I20260924 20:13:06.323920 30082 ts_manager.cc:194] Registered new tserver with Master: 6ccdf27e41e342b6aad9c256c2c8eacd (127.29.83.129:36353)
I20260924 20:13:06.325568 30082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:34145
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:06.346087 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:06.459362 30278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:06.459637 30278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:06.459685 30278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:06.464359 30278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:06.464438 30278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:06.464534 30278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:06.470417 30278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:06.471557 30278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:06.472662 30278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:06.482650 30283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.482643 30284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.482637 30286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:06.483107 30278 server_base.cc:1034] running on GCE node
I20260924 20:13:06.483572 30278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:06.484246 30278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:06.485404 30278 hybrid_clock.cc:648] HybridClock initialized: now 1790280786485351 us; error 66 us; skew 500 ppm
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:06.488524 30278 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:06.489715 30278 webserver.cc:460] Webserver started at http://127.29.83.130:37567/ using document root <none> and password file <none>
I20260924 20:13:06.490337 30278 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:06.490434 30278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:06.490658 30278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:06.492810 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "cd8f0c37555e4a1da96856d52d26cc07"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "8569af3b22a184f5488358d14f65a373"
server_key_iv: "f4dc743e0149f9f55b0eec0618c90c86"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.493408 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "cd8f0c37555e4a1da96856d52d26cc07"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "8569af3b22a184f5488358d14f65a373"
server_key_iv: "f4dc743e0149f9f55b0eec0618c90c86"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.497438 30278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 20:13:06.500133 30293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.501430 30278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:06.501579 30278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "cd8f0c37555e4a1da96856d52d26cc07"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "8569af3b22a184f5488358d14f65a373"
server_key_iv: "f4dc743e0149f9f55b0eec0618c90c86"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.501708 30278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:06.513123 30278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:06.516319 30278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:06.516541 30278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:06.517167 30278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:06.518091 30278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:06.518159 30278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.518234 30278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:06.518276 30278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.527927 30278 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:45505
I20260924 20:13:06.527952 30406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:45505 every 8 connection(s)
I20260924 20:13:06.528861 30278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:06.532531 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30278
I20260924 20:13:06.532677 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:06.532959 30030 external_mini_cluster.cc:1428] Setting key af438511088baedf62a972fb654f8959
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:13:06.545639 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.530967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:46299 (local address 127.29.83.190:46827)
0924 20:13:06.531148 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.531151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.532019 (+ 868us) server_negotiation.cc:407] Connection header received
0924 20:13:06.533534 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.533537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.533595 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.533697 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.535673 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.536398 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.537997 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.538425 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.538558 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.541247 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.541252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.541255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.541290 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.543128 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.543882 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.543888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.543893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.543991 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.544329 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.544331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.544332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.544517 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.544655 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.545323 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.545463 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:13:06.546610 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.531366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46827 (local address 127.29.83.130:46299)
0924 20:13:06.531844 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:06.531889 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:06.533256 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:06.533961 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:06.533971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:06.534554 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:06.535509 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.535525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.536537 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.536542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.537874 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.537884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.538610 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.538638 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.538876 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.539523 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:06.539542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:06.541038 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:06.543288 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.543294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.543311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.543701 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.544096 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.544098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.544100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.544243 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.544741 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:06.544749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:06.545041 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 20:13:06.545282 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 20:13:06.545648 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260924 20:13:06.548321 30407 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46827
I20260924 20:13:06.548583 30407 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:06.549093 30407 heartbeater.cc:511] Master 127.29.83.190:46827 requested a full tablet report, sending...
I20260924 20:13:06.550302 30082 ts_manager.cc:194] Registered new tserver with Master: cd8f0c37555e4a1da96856d52d26cc07 (127.29.83.130:45505)
I20260924 20:13:06.550882 30082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:46299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:06.565588 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:06.680229 30412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:06.680496 30412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:06.680541 30412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:06.684542 30412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:06.684602 30412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:06.684712 30412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:06.690084 30412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:06.691083 30412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:06.692056 30412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:06.701434 30418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.701450 30417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:06.701443 30420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:06.701648 30412 server_base.cc:1034] running on GCE node
I20260924 20:13:06.702219 30412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:06.702788 30412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:06.703949 30412 hybrid_clock.cc:648] HybridClock initialized: now 1790280786703914 us; error 45 us; skew 500 ppm
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:06.707221 30412 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:06.708001 30412 webserver.cc:460] Webserver started at http://127.29.83.131:37497/ using document root <none> and password file <none>
I20260924 20:13:06.708511 30412 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:06.708566 30412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:06.708729 30412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:06.710801 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "4d0d44c510a149bfb5029fb9ab63f681"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "2a4eb5aae91be9390bf63698b9481041"
server_key_iv: "2d4c4279d95605b5bf6c0868999abad6"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.711324 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "4d0d44c510a149bfb5029fb9ab63f681"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "2a4eb5aae91be9390bf63698b9481041"
server_key_iv: "2d4c4279d95605b5bf6c0868999abad6"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.715040 30412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:06.717458 30427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.718868 30412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:06.718967 30412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "4d0d44c510a149bfb5029fb9ab63f681"
format_stamp: "Formatted at 2026-09-24 20:13:06 on dist-test-slave-52fr"
server_key: "2a4eb5aae91be9390bf63698b9481041"
server_key_iv: "2d4c4279d95605b5bf6c0868999abad6"
server_key_version: "encryptionkey@0"
I20260924 20:13:06.719053 30412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:06.745082 30412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:06.748626 30412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:06.748792 30412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:06.749336 30412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:06.750206 30412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:06.750253 30412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.750293 30412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:06.750308 30412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:06.759599 30412 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:32799
I20260924 20:13:06.759621 30540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:32799 every 8 connection(s)
I20260924 20:13:06.760574 30412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:06.762676 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30412
I20260924 20:13:06.762831 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:06.763098 30030 external_mini_cluster.cc:1428] Setting key 00649f80c331c31321dc1cb293623a6b
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:06.775270 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.762548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:50291 (local address 127.29.83.190:46827)
0924 20:13:06.762714 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.762718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.763711 (+ 993us) server_negotiation.cc:407] Connection header received
0924 20:13:06.765142 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.765145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.765199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.765277 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.766559 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.767232 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.768431 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.768668 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.768779 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.771459 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.771463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.771465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.771497 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.773074 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.773604 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.773608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.773610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.773719 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.774107 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.774112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.774113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.774332 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.774446 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.774957 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.775090 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:13:06.776090 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.763064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46827 (local address 127.29.83.131:50291)
0924 20:13:06.763560 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:06.763612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:06.764932 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:06.765453 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:06.765461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:06.765840 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:06.766410 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.766425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.767410 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.767414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.768297 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:06.768307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.768835 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.768839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:06.769010 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.769672 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:06.769688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:06.771284 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:06.773198 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.773204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.773222 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.773490 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.773850 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:06.773853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:06.773855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:06.774005 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:06.774442 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:06.774448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:06.774709 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 20:13:06.774915 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 20:13:06.775256 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":141,"threads_started":1}
I20260924 20:13:06.777290 30541 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46827
I20260924 20:13:06.777599 30541 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:06.778220 30541 heartbeater.cc:511] Master 127.29.83.190:46827 requested a full tablet report, sending...
I20260924 20:13:06.779296 30082 ts_manager.cc:194] Registered new tserver with Master: 4d0d44c510a149bfb5029fb9ab63f681 (127.29.83.131:32799)
I20260924 20:13:06.779840 30082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:50291
I20260924 20:13:06.789574 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:06.816700 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.806611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.29.83.129:36353)
0924 20:13:06.806933 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.806948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.806974 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:13:06.807121 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.807128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.807337 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.807464 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.808212 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.809040 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.810071 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.810519 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.810615 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.813237 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.813250 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.813265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.813292 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.814929 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.815367 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.815373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.815374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.815458 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.815688 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.815690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.815692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.815972 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.816126 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.816324 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.816493 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
W20260924 20:13:06.817818 30187 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33368
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:06.844543 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:06.835742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.29.83.129:36353)
0924 20:13:06.835907 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:06.835913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:06.836086 (+ 173us) server_negotiation.cc:407] Connection header received
0924 20:13:06.836206 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:06.836209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:06.836258 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:06.836328 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:06.836996 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.837547 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.838433 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:06.838681 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:06.838767 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:06.841131 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:06.841134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:06.841137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:06.841161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:06.842773 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.843205 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.843210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.843211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.843298 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:06.843677 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:06.843680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:06.843681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:06.843865 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:06.843994 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:06.844245 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 20:13:06.844369 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:13:06.846009 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30144
I20260924 20:13:06.852037 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30278
I20260924 20:13:06.859799 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30412
I20260924 20:13:06.865549 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30050
2026-09-24T20:13:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2240 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30555](info): set up 2 sockets
Sep 24 20:13:06 dist-test-slave-52fr krb5kdc[30555](info): commencing operation
krb5kdc: starting...
W20260924 20:13:07.901532 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:07 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:08.030257 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:33621
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:33621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:08.144801 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:08.145088 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:08.145133 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:08.149211 30568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:08.149286 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:08.149305 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:08.149322 30568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:08.149336 30568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:08.154770 30568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:33621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:08.155810 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:08.156908 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:08.165859 30568 server_base.cc:1034] running on GCE node
W20260924 20:13:08.165748 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.165786 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.165748 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:08.166678 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:08.167834 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:08.169020 30568 hybrid_clock.cc:648] HybridClock initialized: now 1790280788169007 us; error 27 us; skew 500 ppm
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:08.171958 30568 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:08.172796 30568 webserver.cc:460] Webserver started at http://127.29.83.190:43301/ using document root <none> and password file <none>
I20260924 20:13:08.173316 30568 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:08.173378 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:08.173539 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:08.175589 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "987eed354fa64ed8afe719aa6c7f4872"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "e424e0dbf461165f40979b954f59055e"
server_key_iv: "10cf665a59b968f50eda7a2e4a6d9b19"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.176097 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "987eed354fa64ed8afe719aa6c7f4872"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "e424e0dbf461165f40979b954f59055e"
server_key_iv: "10cf665a59b968f50eda7a2e4a6d9b19"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.179852 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:13:08.182387 30583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.183712 30568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:08.183817 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "987eed354fa64ed8afe719aa6c7f4872"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "e424e0dbf461165f40979b954f59055e"
server_key_iv: "10cf665a59b968f50eda7a2e4a6d9b19"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.183905 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:08.196010 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:08.199242 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:08.199414 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:08.207031 30568 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:33621
I20260924 20:13:08.207053 30635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:33621 every 8 connection(s)
I20260924 20:13:08.208089 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:08.211104 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:08.217147 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872: Bootstrap starting.
I20260924 20:13:08.217227 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30568
I20260924 20:13:08.217343 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:08.217576 30030 external_mini_cluster.cc:1428] Setting key ce0ecaf1de4b3c756abdb1bf65732f74
I20260924 20:13:08.220103 30636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:08.220903 30636 log.cc:824] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:08.223030 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872: No bootstrap required, opened a new log
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:08.227054 30636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } }
I20260924 20:13:08.227361 30636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:08.227450 30636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987eed354fa64ed8afe719aa6c7f4872, State: Initialized, Role: FOLLOWER
I20260924 20:13:08.228044 30636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } }
I20260924 20:13:08.228253 30636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:08.228325 30636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:08.228437 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:08.229804 30636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } }
I20260924 20:13:08.230211 30636 leader_election.cc:302] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987eed354fa64ed8afe719aa6c7f4872; no voters:
I20260924 20:13:08.230603 30636 leader_election.cc:288] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:08.230751 30641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:08.230993 30641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [term 1 LEADER]: Becoming Leader. State: Replica: 987eed354fa64ed8afe719aa6c7f4872, State: Running, Role: LEADER
I20260924 20:13:08.231338 30641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } }
I20260924 20:13:08.232065 30636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:08.233321 30642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987eed354fa64ed8afe719aa6c7f4872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } } }
I20260924 20:13:08.233479 30642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:08.233321 30643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987eed354fa64ed8afe719aa6c7f4872. Latest consensus state: current_term: 1 leader_uuid: "987eed354fa64ed8afe719aa6c7f4872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987eed354fa64ed8afe719aa6c7f4872" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 33621 } } }
I20260924 20:13:08.233538 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.219114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.29.83.190:33621)
0924 20:13:08.219672 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:08.219698 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:08.219736 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:13:08.220534 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:08.220555 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:08.220920 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:08.221309 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:08.222145 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.223174 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.224040 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.224572 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.224768 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.227589 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:08.227596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:08.227615 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:08.227654 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:08.229910 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.230472 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.230479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.230489 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.230668 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.230992 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.230996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.230997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.231517 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:08.231719 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:08.232066 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 20:13:08.232408 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":154,"threads_started":1}
I20260924 20:13:08.233582 30643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:08.234043 30651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:08.237900 30651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:08.244091 30651 catalog_manager.cc:1347] Generated new cluster ID: ae55be5d4c374edda32ab7481e4bcb2b
I20260924 20:13:08.244187 30651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:08.257726 30651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:08.258788 30651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:08.265601 30651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987eed354fa64ed8afe719aa6c7f4872: Generated new TSK 0
I20260924 20:13:08.266454 30651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:08.297114 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:08.411303 30662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:08.411579 30662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:08.411625 30662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:08.416078 30662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:08.416143 30662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:08.416254 30662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:08.422545 30662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:08.423648 30662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:08.424680 30662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:08.434638 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.434643 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.434638 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:08.434937 30662 server_base.cc:1034] running on GCE node
I20260924 20:13:08.435379 30662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:08.436004 30662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:08.437182 30662 hybrid_clock.cc:648] HybridClock initialized: now 1790280788437167 us; error 26 us; skew 500 ppm
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:08.440021 30662 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:08.440923 30662 webserver.cc:460] Webserver started at http://127.29.83.129:33481/ using document root <none> and password file <none>
I20260924 20:13:08.441484 30662 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:08.441543 30662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:08.441826 30662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:08.443879 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "f45662b964b146288a82aed53e40434e"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "530a2efd4bfabd07d06b43d03a95f972"
server_key_iv: "34492d0c9e1a91a243fd0e2d5482dbb8"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.444438 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "f45662b964b146288a82aed53e40434e"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "530a2efd4bfabd07d06b43d03a95f972"
server_key_iv: "34492d0c9e1a91a243fd0e2d5482dbb8"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.448186 30662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:13:08.450616 30677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.452010 30662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:08.452162 30662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "f45662b964b146288a82aed53e40434e"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "530a2efd4bfabd07d06b43d03a95f972"
server_key_iv: "34492d0c9e1a91a243fd0e2d5482dbb8"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.452286 30662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:08.481407 30662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:08.484951 30662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:08.485277 30662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:08.486016 30662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:08.486979 30662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:08.487048 30662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.487118 30662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:08.487171 30662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.496837 30662 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:34911
I20260924 20:13:08.496914 30790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:34911 every 8 connection(s)
I20260924 20:13:08.497947 30662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:08.504164 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30662
I20260924 20:13:08.504311 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:08.504565 30030 external_mini_cluster.cc:1428] Setting key 792004d761d0972dfa4169fa10bfd358
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:08.513919 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.500077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:38843 (local address 127.29.83.190:33621)
0924 20:13:08.500250 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:08.500254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:08.501187 (+ 933us) server_negotiation.cc:407] Connection header received
0924 20:13:08.502558 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:08.502562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:08.502625 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:08.502762 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:08.504113 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.504780 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.506192 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.506440 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.506554 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.509484 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:08.509490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:08.509497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:08.509532 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:08.511337 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.511958 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.511963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.511965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.512058 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.512618 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.512622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.512624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.512850 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:08.512970 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:08.513547 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 20:13:08.513715 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:13:08.514863 30793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.500470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:33621 (local address 127.29.83.129:38843)
0924 20:13:08.500986 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:08.501033 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:08.502317 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:08.502950 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:08.502957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:08.503420 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:08.503946 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.503961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.504908 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.504918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.506061 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.506070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.506594 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.506598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.506784 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.507689 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:08.507706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:08.509258 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:08.511480 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.511487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.511503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.511804 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.512165 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.512171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.512174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.512477 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.512968 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:08.512976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:08.513289 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 20:13:08.513507 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 20:13:08.513945 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:08.516251 30791 heartbeater.cc:347] Connected to a master server at 127.29.83.190:33621
I20260924 20:13:08.516520 30791 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:08.517083 30791 heartbeater.cc:511] Master 127.29.83.190:33621 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:13:08.518970 30600 ts_manager.cc:194] Registered new tserver with Master: f45662b964b146288a82aed53e40434e (127.29.83.129:34911)
I20260924 20:13:08.520406 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:38843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:08.538338 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:08.657100 30796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:08.657444 30796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:08.657531 30796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:08.661815 30796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:08.661936 30796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:08.662046 30796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:08.668073 30796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:08.669540 30796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:08.670686 30796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:08.681304 30802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.681396 30801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.681424 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:08.681504 30796 server_base.cc:1034] running on GCE node
I20260924 20:13:08.682137 30796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:08.682860 30796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:08.684083 30796 hybrid_clock.cc:648] HybridClock initialized: now 1790280788684051 us; error 46 us; skew 500 ppm
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:08.687269 30796 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:08.688298 30796 webserver.cc:460] Webserver started at http://127.29.83.130:37689/ using document root <none> and password file <none>
I20260924 20:13:08.688942 30796 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:08.689033 30796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:08.689260 30796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:08.691438 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "283d222e178d40b68539f46f109b7fe9"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "d9516c51ce0c63cb354de30fcd2678e3"
server_key_iv: "2ced588b8493b082a3dd678083dc5cef"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.692025 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "283d222e178d40b68539f46f109b7fe9"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "d9516c51ce0c63cb354de30fcd2678e3"
server_key_iv: "2ced588b8493b082a3dd678083dc5cef"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.695962 30796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:08.698726 30811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.700052 30796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:08.700219 30796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "283d222e178d40b68539f46f109b7fe9"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "d9516c51ce0c63cb354de30fcd2678e3"
server_key_iv: "2ced588b8493b082a3dd678083dc5cef"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.700351 30796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:08.722352 30796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:08.725634 30796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:08.725878 30796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:08.726475 30796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:08.727312 30796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:08.727384 30796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.727458 30796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:08.727509 30796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.737172 30796 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:41783
I20260924 20:13:08.737202 30924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:41783 every 8 connection(s)
I20260924 20:13:08.738185 30796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:08.745452 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30796
I20260924 20:13:08.745568 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:08.745900 30030 external_mini_cluster.cc:1428] Setting key f37b467be42649e11f67c925e70c52c9
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:08.753161 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.740119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:34899 (local address 127.29.83.190:33621)
0924 20:13:08.740273 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:08.740276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:08.741125 (+ 849us) server_negotiation.cc:407] Connection header received
0924 20:13:08.742461 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:08.742464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:08.742514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:08.742605 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:08.743847 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.744404 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.745842 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.746115 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.746240 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.749363 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:08.749366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:08.749368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:08.749395 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:08.750997 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.751572 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.751576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.751578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.751663 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.752027 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.752030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.752032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.752231 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:08.752342 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:08.752871 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 20:13:08.753001 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:13:08.754117 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.740508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:33621 (local address 127.29.83.130:34899)
0924 20:13:08.740958 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:08.741003 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:08.742238 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:08.742777 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:08.742784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:08.743137 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:08.743660 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.743674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.744533 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.744536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.745654 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.745668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.746311 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.746317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.746498 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.747263 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:08.747280 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:08.749136 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:08.751131 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.751137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.751155 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.751444 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.751766 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.751770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.751772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.751922 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.752339 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:08.752347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:08.752619 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 20:13:08.752829 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 20:13:08.753185 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:08.755342 30925 heartbeater.cc:347] Connected to a master server at 127.29.83.190:33621
I20260924 20:13:08.755620 30925 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:08.756124 30925 heartbeater.cc:511] Master 127.29.83.190:33621 requested a full tablet report, sending...
I20260924 20:13:08.757340 30600 ts_manager.cc:194] Registered new tserver with Master: 283d222e178d40b68539f46f109b7fe9 (127.29.83.130:41783)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:13:08.758143 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:34899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:08.779029 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:08.895085 30930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:08.895382 30930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:08.895428 30930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:08.899542 30930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:08.899613 30930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:08.899677 30930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:08.905288 30930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:33621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:08.906400 30930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:08.907411 30930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:08.917927 30938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:08.917927 30936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:08.917987 30930 server_base.cc:1034] running on GCE node
W20260924 20:13:08.917937 30935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:08.918649 30930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:08.919382 30930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:08.920598 30930 hybrid_clock.cc:648] HybridClock initialized: now 1790280788920573 us; error 37 us; skew 500 ppm
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:08.923648 30930 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:08.924553 30930 webserver.cc:460] Webserver started at http://127.29.83.131:42149/ using document root <none> and password file <none>
I20260924 20:13:08.925123 30930 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:08.925182 30930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:08.925349 30930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:08.927455 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "5e94335530b34515813be0e03ab791ce"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "fc4bdf3d15aff44b8bf2a4115bf51b65"
server_key_iv: "7a77e79add6c9d711fec36accb30046a"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.927994 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "5e94335530b34515813be0e03ab791ce"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "fc4bdf3d15aff44b8bf2a4115bf51b65"
server_key_iv: "7a77e79add6c9d711fec36accb30046a"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.932178 30930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 20:13:08.935372 30945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.936858 30930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:13:08.936990 30930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "5e94335530b34515813be0e03ab791ce"
format_stamp: "Formatted at 2026-09-24 20:13:08 on dist-test-slave-52fr"
server_key: "fc4bdf3d15aff44b8bf2a4115bf51b65"
server_key_iv: "7a77e79add6c9d711fec36accb30046a"
server_key_version: "encryptionkey@0"
I20260924 20:13:08.937091 30930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:08.954324 30930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:08.958312 30930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:08.958539 30930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:08.959148 30930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:08.960011 30930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:08.960083 30930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.960155 30930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:08.960207 30930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:08.969123 30930 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:43835
I20260924 20:13:08.969190 31058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:43835 every 8 connection(s)
I20260924 20:13:08.970206 30930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:08.976140 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 30930
I20260924 20:13:08.976302 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:08.976580 30030 external_mini_cluster.cc:1428] Setting key d661f5173f85de61a1d88e3b71df314f
Sep 24 20:13:08 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280788, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:08.985059 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.972286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:58039 (local address 127.29.83.190:33621)
0924 20:13:08.972437 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:08.972441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:08.973317 (+ 876us) server_negotiation.cc:407] Connection header received
0924 20:13:08.974786 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:08.974790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:08.974853 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:08.974952 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:08.976262 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.976965 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.978704 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.979043 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.979130 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.981649 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:08.981652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:08.981654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:08.981679 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:08.983194 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.983673 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.983676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.983677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.983777 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:08.984076 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:08.984078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:08.984079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:08.984257 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:08.984332 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:08.984832 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 20:13:08.984933 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:13:08.986058 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.972693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:33621 (local address 127.29.83.131:58039)
0924 20:13:08.973155 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:08.973201 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:08.974549 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:08.975156 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:08.975164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:08.975557 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:08.976088 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.976104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.977096 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.977099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.978561 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:08.978573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.979250 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.979254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:08.979411 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.979975 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:08.979991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:08.981481 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:08.983301 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.983306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.983326 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.983588 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.983860 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:08.983862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:08.983863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:08.983999 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:08.984354 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:08.984360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:08.984597 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 20:13:08.984756 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 20:13:08.985018 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260924 20:13:08.987356 31059 heartbeater.cc:347] Connected to a master server at 127.29.83.190:33621
I20260924 20:13:08.987630 31059 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:08.988194 31059 heartbeater.cc:511] Master 127.29.83.190:33621 requested a full tablet report, sending...
I20260924 20:13:08.989383 30600 ts_manager.cc:194] Registered new tserver with Master: 5e94335530b34515813be0e03ab791ce (127.29.83.131:43835)
I20260924 20:13:08.990262 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:58039
I20260924 20:13:08.991074 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:13:09.001682 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:08.994376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.29.83.190:33621)
0924 20:13:08.994506 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:08.994509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:08.994741 (+ 232us) server_negotiation.cc:407] Connection header received
0924 20:13:08.994862 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:08.994864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:08.994901 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:08.994968 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:08.995836 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.996342 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.997189 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:08.997399 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:08.997475 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:08.998902 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:08.998905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:08.998907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:08.998926 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:09.000165 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.000569 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.000572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.000573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.000642 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.000900 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.000903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.000904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.001115 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:09.001219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.001463 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.001557 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:09.006989 30599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:09.009922 30599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:09.025425 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.019678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:41783 (local address 127.0.0.1:36128)
0924 20:13:09.020607 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:09.020621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:09.020739 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:09.021169 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:09.021172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:09.021192 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:09.021310 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.021318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022445 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.022450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.023785 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.023795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.024859 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.024863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.024965 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.025001 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 20:13:09.025110 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 20:13:09.025254 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260924 20:13:09.025594 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.018961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.29.83.131:43835)
0924 20:13:09.019712 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.019723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.019982 (+ 259us) server_negotiation.cc:407] Connection header received
0924 20:13:09.020285 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.020291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.020504 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.020634 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:09.021289 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022510 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.024189 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.024742 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.024850 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.024997 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.025330 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.025430 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":94,"threads_started":1}
I20260924 20:13:09.025637 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.019174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.29.83.129:34911)
0924 20:13:09.019570 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.019579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.019983 (+ 404us) server_negotiation.cc:407] Connection header received
0924 20:13:09.020285 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.020291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.020504 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.020626 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:09.021289 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022518 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.024210 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.024995 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.025119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.025261 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.025371 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.025481 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":93,"threads_started":1}
I20260924 20:13:09.026232 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.019917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.29.83.130:41783)
0924 20:13:09.020338 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.020357 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.020713 (+ 356us) server_negotiation.cc:407] Connection header received
0924 20:13:09.020859 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.020865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.021023 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.021185 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:09.021434 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022294 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.023911 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.024712 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.024842 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.024999 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.026013 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.026127 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260924 20:13:09.025636 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.018889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34911 (local address 127.0.0.1:44278)
0924 20:13:09.019834 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:09.019892 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:09.020103 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:09.020843 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:09.020849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:09.020885 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:09.021153 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.021160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022665 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.022668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.024072 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.024082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.025122 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.025126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.025225 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.025253 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:13:09.025334 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:13:09.025423 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":200,"spinlock_wait_cycles":13824,"thread_start_us":86,"threads_started":1}
I20260924 20:13:09.026041 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.018866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:43835 (local address 127.0.0.1:60724)
0924 20:13:09.019834 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:09.019892 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:09.020103 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:09.020843 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:09.020849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:09.020883 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:09.021152 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.021160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.022665 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.022668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.024042 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.024053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.025092 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.025098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.025183 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.025203 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:13:09.025807 (+ 604us) client_negotiation.cc:240] Negotiation successful
0924 20:13:09.025900 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":211,"spinlock_wait_cycles":7040,"thread_start_us":113,"threads_started":1}
I20260924 20:13:09.029635 30993 tablet_service.cc:1467] Processing CreateTablet for tablet e85808b96925473a8cd52567061141b3 (DEFAULT_TABLE table=test-table [id=26683a084e934c4f8c411cbd57d0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:09.029661 30859 tablet_service.cc:1467] Processing CreateTablet for tablet e85808b96925473a8cd52567061141b3 (DEFAULT_TABLE table=test-table [id=26683a084e934c4f8c411cbd57d0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:09.029660 30725 tablet_service.cc:1467] Processing CreateTablet for tablet e85808b96925473a8cd52567061141b3 (DEFAULT_TABLE table=test-table [id=26683a084e934c4f8c411cbd57d0edba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:09.030761 30993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85808b96925473a8cd52567061141b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:09.030778 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85808b96925473a8cd52567061141b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:09.030778 30725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85808b96925473a8cd52567061141b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:09.037483 31076 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Bootstrap starting.
I20260924 20:13:09.039479 31077 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Bootstrap starting.
I20260924 20:13:09.039917 31076 tablet_bootstrap.cc:654] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:09.040665 31078 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Bootstrap starting.
I20260924 20:13:09.041384 31076 log.cc:824] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:09.042431 31077 tablet_bootstrap.cc:654] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:09.043102 31078 tablet_bootstrap.cc:654] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:09.043210 31077 log.cc:824] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:09.043728 31076 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: No bootstrap required, opened a new log
I20260924 20:13:09.043889 31078 log.cc:824] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:09.043942 31076 ts_tablet_manager.cc:1397] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260924 20:13:09.045387 31077 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: No bootstrap required, opened a new log
I20260924 20:13:09.045599 31077 ts_tablet_manager.cc:1397] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:13:09.047241 31078 tablet_bootstrap.cc:492] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: No bootstrap required, opened a new log
I20260924 20:13:09.047442 31078 ts_tablet_manager.cc:1397] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 20:13:09.047639 31076 raft_consensus.cc:348] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.047956 31076 raft_consensus.cc:374] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:09.048024 31076 raft_consensus.cc:729] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e94335530b34515813be0e03ab791ce, State: Initialized, Role: FOLLOWER
I20260924 20:13:09.048512 31076 consensus_queue.cc:260] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.049265 31059 heartbeater.cc:503] Master 127.29.83.190:33621 was elected leader, sending a full tablet report...
I20260924 20:13:09.049825 31077 raft_consensus.cc:348] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.050153 31076 ts_tablet_manager.cc:1428] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260924 20:13:09.050258 31077 raft_consensus.cc:374] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:09.050369 31077 raft_consensus.cc:729] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283d222e178d40b68539f46f109b7fe9, State: Initialized, Role: FOLLOWER
I20260924 20:13:09.050882 31077 consensus_queue.cc:260] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.051630 30925 heartbeater.cc:503] Master 127.29.83.190:33621 was elected leader, sending a full tablet report...
I20260924 20:13:09.052066 31077 ts_tablet_manager.cc:1428] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260924 20:13:09.053997 31078 raft_consensus.cc:348] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.054294 31078 raft_consensus.cc:374] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:09.054370 31078 raft_consensus.cc:729] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45662b964b146288a82aed53e40434e, State: Initialized, Role: FOLLOWER
I20260924 20:13:09.054880 31078 consensus_queue.cc:260] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.055744 30791 heartbeater.cc:503] Master 127.29.83.190:33621 was elected leader, sending a full tablet report...
I20260924 20:13:09.056046 31078 ts_tablet_manager.cc:1428] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
W20260924 20:13:09.221259 31060 tablet.cc:2369] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:09.239413 30926 tablet.cc:2369] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:09.249739 30792 tablet.cc:2369] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:09.395995 31083 raft_consensus.cc:482] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:09.396286 31083 raft_consensus.cc:504] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.397526 31083 leader_election.cc:288] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e94335530b34515813be0e03ab791ce (127.29.83.131:43835), f45662b964b146288a82aed53e40434e (127.29.83.129:34911)
I20260924 20:13:09.401573 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.398021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:49979 (local address 127.29.83.131:43835)
0924 20:13:09.398224 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.398231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.398328 (+ 97us) server_negotiation.cc:407] Connection header received
0924 20:13:09.398488 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.398496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.398534 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.398616 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:09.398938 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.399474 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.400580 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.401019 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.401155 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.401212 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.401379 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.401446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 20:13:09.401594 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.397894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:43835 (local address 127.29.83.130:49979)
0924 20:13:09.398222 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:09.398237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:09.398373 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:09.398635 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:09.398638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:09.398653 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:09.398826 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.398831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.399584 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.399588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.400453 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.400461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.401141 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.401144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.401244 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.401278 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 20:13:09.401365 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:13:09.401439 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":56,"thread_start_us":66,"threads_started":1}
I20260924 20:13:09.401595 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.397994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:51951 (local address 127.29.83.129:34911)
0924 20:13:09.398155 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.398160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.398305 (+ 145us) server_negotiation.cc:407] Connection header received
0924 20:13:09.398480 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.398482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.398531 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.398614 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:09.398958 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.399452 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.400603 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.401029 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.401161 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.401217 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.401401 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.401477 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 20:13:09.401616 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.397894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34911 (local address 127.29.83.130:51951)
0924 20:13:09.398187 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:09.398203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:09.398357 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:09.398626 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:09.398629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:09.398644 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:09.398838 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.398844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.399567 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.399571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.400477 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:09.400486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.401158 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.401162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:09.401271 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.401291 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:13:09.401387 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 20:13:09.401457 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":95,"thread_start_us":72,"threads_started":1}
I20260924 20:13:09.402323 31013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85808b96925473a8cd52567061141b3" candidate_uuid: "283d222e178d40b68539f46f109b7fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e94335530b34515813be0e03ab791ce" is_pre_election: true
I20260924 20:13:09.402482 30745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85808b96925473a8cd52567061141b3" candidate_uuid: "283d222e178d40b68539f46f109b7fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45662b964b146288a82aed53e40434e" is_pre_election: true
I20260924 20:13:09.402649 31013 raft_consensus.cc:2454] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283d222e178d40b68539f46f109b7fe9 in term 0.
I20260924 20:13:09.402793 30745 raft_consensus.cc:2454] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283d222e178d40b68539f46f109b7fe9 in term 0.
I20260924 20:13:09.403199 30813 leader_election.cc:302] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 283d222e178d40b68539f46f109b7fe9, 5e94335530b34515813be0e03ab791ce, f45662b964b146288a82aed53e40434e; no voters:
I20260924 20:13:09.403445 31083 raft_consensus.cc:2781] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:09.403546 31083 raft_consensus.cc:482] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:09.403605 31083 raft_consensus.cc:3037] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:09.404778 31083 raft_consensus.cc:504] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.405205 31083 leader_election.cc:288] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [CANDIDATE]: Term 1 election: Requested vote from peers 5e94335530b34515813be0e03ab791ce (127.29.83.131:43835), f45662b964b146288a82aed53e40434e (127.29.83.129:34911)
I20260924 20:13:09.405591 30745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85808b96925473a8cd52567061141b3" candidate_uuid: "283d222e178d40b68539f46f109b7fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45662b964b146288a82aed53e40434e"
I20260924 20:13:09.405684 31013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85808b96925473a8cd52567061141b3" candidate_uuid: "283d222e178d40b68539f46f109b7fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e94335530b34515813be0e03ab791ce"
I20260924 20:13:09.405730 30745 raft_consensus.cc:3037] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:09.405872 31013 raft_consensus.cc:3037] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:09.406886 30745 raft_consensus.cc:2454] T e85808b96925473a8cd52567061141b3 P f45662b964b146288a82aed53e40434e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283d222e178d40b68539f46f109b7fe9 in term 1.
I20260924 20:13:09.406890 31013 raft_consensus.cc:2454] T e85808b96925473a8cd52567061141b3 P 5e94335530b34515813be0e03ab791ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283d222e178d40b68539f46f109b7fe9 in term 1.
I20260924 20:13:09.407219 30814 leader_election.cc:302] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283d222e178d40b68539f46f109b7fe9, f45662b964b146288a82aed53e40434e; no voters:
I20260924 20:13:09.407392 31083 raft_consensus.cc:2781] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:09.407620 31083 raft_consensus.cc:686] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 283d222e178d40b68539f46f109b7fe9, State: Running, Role: LEADER
I20260924 20:13:09.407969 31083 consensus_queue.cc:237] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } }
I20260924 20:13:09.411381 30599 catalog_manager.cc:5495] T e85808b96925473a8cd52567061141b3 P 283d222e178d40b68539f46f109b7fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 283d222e178d40b68539f46f109b7fe9 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "283d222e178d40b68539f46f109b7fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e94335530b34515813be0e03ab791ce" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45662b964b146288a82aed53e40434e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283d222e178d40b68539f46f109b7fe9" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41783 } health_report { overall_health: HEALTHY } } }
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:09.494549 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.477860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.29.83.129:34911)
0924 20:13:09.478035 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.478040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.478218 (+ 178us) server_negotiation.cc:407] Connection header received
0924 20:13:09.478394 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.478398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.478449 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.478547 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:09.479266 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.479804 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.480678 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.480927 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.481044 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.490520 (+ 9476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:09.490537 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:09.490553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:09.490590 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:09.492464 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.493059 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.493066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.493067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.493173 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.493477 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.493492 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.493494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.493840 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:09.494083 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.494243 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.494377 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:09.525892 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.516332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.29.83.129:34911)
0924 20:13:09.516476 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.516481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.516685 (+ 204us) server_negotiation.cc:407] Connection header received
0924 20:13:09.516863 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.516870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.516915 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.517008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:09.517715 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.518314 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.519183 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.519424 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.519533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.522544 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:09.522548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:09.522550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:09.522578 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:09.524081 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.524534 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.524538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.524539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.524633 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.524915 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.524919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.524921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.525102 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:09.525252 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.525569 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.525701 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:09.553730 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:09.543863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.29.83.129:34911)
0924 20:13:09.544041 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:09.544048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:09.544554 (+ 506us) server_negotiation.cc:407] Connection header received
0924 20:13:09.544744 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:09.544748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:09.544823 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:09.544924 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:09.545775 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.546381 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.547247 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:09.547457 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:09.547535 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:09.550201 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:09.550204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:09.550206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:09.550246 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:09.551786 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.552357 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.552359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.552361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.552439 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:09.552827 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:09.552830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:09.552831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:09.553031 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:09.553148 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:09.553408 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 20:13:09.553515 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:13:09.555109 30705 tablet_service.cc:2908] Rejecting scan request for tablet e85808b96925473a8cd52567061141b3: Uninitialized: clean time has not yet been initialized
I20260924 20:13:09.557248 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30662
I20260924 20:13:09.563170 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30796
I20260924 20:13:09.568626 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30930
I20260924 20:13:09.574244 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 30568
2026-09-24T20:13:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2708 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[31101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[31101](info): set up 2 sockets
Sep 24 20:13:09 dist-test-slave-52fr krb5kdc[31101](info): commencing operation
krb5kdc: starting...
W20260924 20:13:10.629528 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:10 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:10.757200 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:37595
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:10.868603 31114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:10.868904 31114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:10.868947 31114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:10.873085 31114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:10.873148 31114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:13:10.873167 31114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:10.873185 31114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:10.873201 31114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:10.873238 31114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:10.879387 31114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:37595
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:10.880578 31114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:10.881675 31114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:10.890825 31119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:10.890829 31122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:10.890947 31114 server_base.cc:1034] running on GCE node
W20260924 20:13:10.890834 31120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:10.891743 31114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:10.892879 31114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:10.894102 31114 hybrid_clock.cc:648] HybridClock initialized: now 1790280790894081 us; error 29 us; skew 500 ppm
Sep 24 20:13:10 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280790, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:10.897019 31114 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:10.898027 31114 webserver.cc:460] Webserver started at http://127.29.83.190:35701/ using document root <none> and password file <none>
I20260924 20:13:10.898593 31114 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:10.898680 31114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:10.898998 31114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:10.901085 31114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "3986d68d88524d81b9f1c7d9621fe004"
format_stamp: "Formatted at 2026-09-24 20:13:10 on dist-test-slave-52fr"
server_key: "fb9b241d0ee162b187fd9a3e1aaee580"
server_key_iv: "07ccf3073acdf8ca346e51537e22dacb"
server_key_version: "encryptionkey@0"
I20260924 20:13:10.901655 31114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "3986d68d88524d81b9f1c7d9621fe004"
format_stamp: "Formatted at 2026-09-24 20:13:10 on dist-test-slave-52fr"
server_key: "fb9b241d0ee162b187fd9a3e1aaee580"
server_key_iv: "07ccf3073acdf8ca346e51537e22dacb"
server_key_version: "encryptionkey@0"
I20260924 20:13:10.905514 31114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 20:13:10.908058 31129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:10.909406 31114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:10.909658 31114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "3986d68d88524d81b9f1c7d9621fe004"
format_stamp: "Formatted at 2026-09-24 20:13:10 on dist-test-slave-52fr"
server_key: "fb9b241d0ee162b187fd9a3e1aaee580"
server_key_iv: "07ccf3073acdf8ca346e51537e22dacb"
server_key_version: "encryptionkey@0"
I20260924 20:13:10.909798 31114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:10.931771 31114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:10.935214 31114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:10.935454 31114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:10.943399 31114 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:37595
I20260924 20:13:10.943425 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:37595 every 8 connection(s)
I20260924 20:13:10.944386 31114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:10.947256 31192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:10.952785 31192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Bootstrap starting.
I20260924 20:13:10.953884 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31114
I20260924 20:13:10.954025 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:10.954294 30030 external_mini_cluster.cc:1428] Setting key d1b10e3724cb489badd7b0143084cfaa
I20260924 20:13:10.955740 31192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:10.956499 31192 log.cc:824] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:10.958549 31192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: No bootstrap required, opened a new log
I20260924 20:13:10.962455 31192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } }
I20260924 20:13:10.962730 31192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:10.962800 31192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986d68d88524d81b9f1c7d9621fe004, State: Initialized, Role: FOLLOWER
Sep 24 20:13:10 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:10.963325 31192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } }
I20260924 20:13:10.963495 31192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:10.963574 31192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:10.963682 31192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:10.964812 31192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } }
I20260924 20:13:10.965165 31192 leader_election.cc:302] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986d68d88524d81b9f1c7d9621fe004; no voters:
I20260924 20:13:10.965503 31192 leader_election.cc:288] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:10.965621 31197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:10.965866 31197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [term 1 LEADER]: Becoming Leader. State: Replica: 3986d68d88524d81b9f1c7d9621fe004, State: Running, Role: LEADER
I20260924 20:13:10.966246 31197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } }
I20260924 20:13:10.966945 31192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:10.968303 31199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3986d68d88524d81b9f1c7d9621fe004. Latest consensus state: current_term: 1 leader_uuid: "3986d68d88524d81b9f1c7d9621fe004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } } }
I20260924 20:13:10.968430 31199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:10.969324 31198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3986d68d88524d81b9f1c7d9621fe004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986d68d88524d81b9f1c7d9621fe004" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 37595 } } }
I20260924 20:13:10.969554 31198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:10.969853 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:10.955495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.29.83.190:37595)
0924 20:13:10.956021 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:10.956036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:10.956070 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:13:10.956788 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:10.956806 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:10.957247 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:10.957604 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:10.958467 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:10.959481 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:10.960527 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:10.961027 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:10.961207 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:10.963939 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:10.963944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:10.963958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:10.963984 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:10.965960 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:10.966484 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:10.966490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:10.966496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:10.966632 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:10.966947 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:10.966950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:10.966952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:10.967486 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:10.967697 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:10.968426 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 20:13:10.968741 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":178,"threads_started":1}
W20260924 20:13:10.972326 31213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:10.972415 31213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:10.972498 31206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:10.973248 31206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:10.978892 31206 catalog_manager.cc:1347] Generated new cluster ID: 349487aa7b924e1587986500a9754409
I20260924 20:13:10.978971 31206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:10.985687 31206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:10.986907 31206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:10.998653 31206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Generated new TSK 0
I20260924 20:13:10.999541 31206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:11.033582 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:37595
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:11.170125 31218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:11.170415 31218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:11.170462 31218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:11.174533 31218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:11.174595 31218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:11.174664 31218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:11.174731 31218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:11.180137 31218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:11.181201 31218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:11.182250 31218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:11.191920 31223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:11.191915 31226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:11.191915 31224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:11.192294 31218 server_base.cc:1034] running on GCE node
I20260924 20:13:11.192732 31218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:11.193446 31218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:11.194638 31218 hybrid_clock.cc:648] HybridClock initialized: now 1790280791194588 us; error 61 us; skew 500 ppm
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:11.197573 31218 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:11.198633 31218 webserver.cc:460] Webserver started at http://127.29.83.129:35277/ using document root <none> and password file <none>
I20260924 20:13:11.199239 31218 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:11.199337 31218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:11.199564 31218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:11.201666 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "f3569de8895b471681fd115a1e980a7e"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "7e4626046a753b189e06aafd9d2ad1e2"
server_key_iv: "c16eedc013ab007f72fc19a6d4f55e1f"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.202332 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "f3569de8895b471681fd115a1e980a7e"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "7e4626046a753b189e06aafd9d2ad1e2"
server_key_iv: "c16eedc013ab007f72fc19a6d4f55e1f"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.206305 31218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:13:11.208988 31233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.210441 31218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:11.210589 31218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "f3569de8895b471681fd115a1e980a7e"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "7e4626046a753b189e06aafd9d2ad1e2"
server_key_iv: "c16eedc013ab007f72fc19a6d4f55e1f"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.210724 31218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:11.231266 31218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:11.234710 31218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:11.234977 31218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:11.236670 31218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:11.236752 31218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.236832 31218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:11.236876 31218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:11.249130 31218 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:46563
I20260924 20:13:11.249186 31348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:46563 every 8 connection(s)
I20260924 20:13:11.250416 31218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:11.251492 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31218
I20260924 20:13:11.251609 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:11.251890 30030 external_mini_cluster.cc:1428] Setting key 546c0c2e405f1132b42c80d7b700fbc8
I20260924 20:13:11.252679 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.237444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:52821 (local address 127.29.83.190:37595)
0924 20:13:11.237614 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.237620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.238990 (+ 1370us) server_negotiation.cc:407] Connection header received
0924 20:13:11.240472 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.240477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.240543 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.240636 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.242175 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.242731 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.244076 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.244455 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.244573 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.247906 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.247913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.247918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.248055 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.250052 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.250713 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.250719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.250721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.250835 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.251311 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.251315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.251317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.251601 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.251715 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.252353 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.252526 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:13:11.253633 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.238319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.129:52821)
0924 20:13:11.238813 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.238874 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.240166 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.240853 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.240863 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.241390 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.242015 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.242033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.242890 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.242894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.243945 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.243956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.244622 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.244626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.244809 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.245473 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.245501 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.247676 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.250219 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.250224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.250237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.250559 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.250999 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.251003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.251005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.251181 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.251776 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.251782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.252071 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.252276 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.252643 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
I20260924 20:13:11.255092 31349 heartbeater.cc:347] Connected to a master server at 127.29.83.190:37595
I20260924 20:13:11.255263 31349 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:11.255829 31349 heartbeater.cc:511] Master 127.29.83.190:37595 requested a full tablet report, sending...
I20260924 20:13:11.257426 31145 ts_manager.cc:194] Registered new tserver with Master: f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563)
I20260924 20:13:11.258950 31145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:52821
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:11.265928 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.257170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.129:52383)
0924 20:13:11.257616 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.257632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.257730 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.258074 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.258077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.258307 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.258425 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.258432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.259619 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.259625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.260478 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.260488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.260976 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.260979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.261067 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.261619 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.261639 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.262180 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.264120 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.264125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.264130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.264384 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.264741 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.264745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.264747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.264839 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.265313 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.265316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.265463 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.265578 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.265695 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
I20260924 20:13:11.265944 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.257238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:52383 (local address 127.29.83.190:37595)
0924 20:13:11.257381 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.257387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.257713 (+ 326us) server_negotiation.cc:407] Connection header received
0924 20:13:11.257893 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.257899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.257958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.258046 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.258553 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.259457 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.260611 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.260857 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.260949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.262359 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.262362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.262364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.262388 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.263994 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.264515 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.264519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.264525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.264641 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.264960 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.264964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.264965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.265173 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.265292 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.265652 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.265816 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:11.282037 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:37595
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:11.395454 31358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:11.395759 31358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:11.395836 31358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:11.399976 31358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:11.400064 31358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:11.400132 31358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:11.400216 31358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:11.405558 31358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:11.406729 31358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:11.407811 31358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:11.418311 31364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:11.418440 31358 server_base.cc:1034] running on GCE node
W20260924 20:13:11.418311 31366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:11.418694 31363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:11.419139 31358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:11.419811 31358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:11.421046 31358 hybrid_clock.cc:648] HybridClock initialized: now 1790280791421021 us; error 34 us; skew 500 ppm
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:11.423987 31358 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:11.424889 31358 webserver.cc:460] Webserver started at http://127.29.83.130:44651/ using document root <none> and password file <none>
I20260924 20:13:11.425462 31358 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:11.425523 31358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:11.425799 31358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:11.427896 31358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "6a30838ad913433dbff428455be70825"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "a5cdef0ce5be65ff587f7259b33722a3"
server_key_iv: "70d27f52062b9c20b8e5b5590a27ce3b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.428452 31358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "6a30838ad913433dbff428455be70825"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "a5cdef0ce5be65ff587f7259b33722a3"
server_key_iv: "70d27f52062b9c20b8e5b5590a27ce3b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.432293 31358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:13:11.434880 31373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.436084 31358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:11.436206 31358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "6a30838ad913433dbff428455be70825"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "a5cdef0ce5be65ff587f7259b33722a3"
server_key_iv: "70d27f52062b9c20b8e5b5590a27ce3b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.436352 31358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:11.458927 31358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:11.462255 31358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:11.462464 31358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:11.464358 31358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:11.464433 31358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.464501 31358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:11.464541 31358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:11.475311 31358 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:33549
I20260924 20:13:11.475366 31488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:33549 every 8 connection(s)
I20260924 20:13:11.476517 31358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:11.478271 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31358
I20260924 20:13:11.478389 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:11.478663 30030 external_mini_cluster.cc:1428] Setting key 8fe7c526cf944fd572555873991d0889
I20260924 20:13:11.479346 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.465289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:60207 (local address 127.29.83.190:37595)
0924 20:13:11.465453 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.465458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.466400 (+ 942us) server_negotiation.cc:407] Connection header received
0924 20:13:11.467663 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.467667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.467739 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.467855 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.469461 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.470201 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.471358 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.471591 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.471711 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.474974 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.474979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.474981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.475018 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.476783 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.477526 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.477531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.477533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.477636 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.478050 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.478054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.478055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.478256 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.478387 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.479034 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.479175 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:13:11.480255 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.465720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.130:60207)
0924 20:13:11.466227 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.466279 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.467396 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.468070 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.468081 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.468633 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.469250 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.469272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.470322 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.470325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.471233 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.471242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.471772 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.471776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.471947 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.473119 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.473142 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.474747 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.476987 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.476993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.477010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.477390 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.477751 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.477754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.477791 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.477936 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.478376 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.478383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.478707 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.478913 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.479238 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260924 20:13:11.481608 31489 heartbeater.cc:347] Connected to a master server at 127.29.83.190:37595
I20260924 20:13:11.481839 31489 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:11.482375 31489 heartbeater.cc:511] Master 127.29.83.190:37595 requested a full tablet report, sending...
I20260924 20:13:11.483510 31146 ts_manager.cc:194] Registered new tserver with Master: 6a30838ad913433dbff428455be70825 (127.29.83.130:33549)
I20260924 20:13:11.484359 31146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:60207
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:11.492528 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.483829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:60841 (local address 127.29.83.190:37595)
0924 20:13:11.483942 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.483947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.484299 (+ 352us) server_negotiation.cc:407] Connection header received
0924 20:13:11.484437 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.484440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.484489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.484569 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.485073 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.485985 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.486935 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.487367 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.487503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.488805 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.488810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.488813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.488849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.490708 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.491213 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.491217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.491219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.491333 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.491634 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.491636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.491637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.491873 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.491971 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.492280 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.492404 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:13:11.492550 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.483934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.130:60841)
0924 20:13:11.484218 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.484231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.484342 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.484584 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.484586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.484820 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.484945 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.484952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.486132 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.486135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.486805 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.486814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.487583 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.487588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.487699 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.488293 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.488307 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.488647 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.490842 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.490846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.490847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.491070 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.491446 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.491449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.491451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.491544 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.491974 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.491976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.492071 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.492159 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.492408 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:11.509389 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:37595
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:11.624640 31498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:11.624910 31498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:11.624991 31498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:11.629034 31498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:11.629098 31498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:11.629137 31498 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:11.629210 31498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:11.634513 31498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:11.635586 31498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:11.636600 31498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:11.646656 31498 server_base.cc:1034] running on GCE node
W20260924 20:13:11.646677 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:11.646667 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:11.646832 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:11.647274 31498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:11.647964 31498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:11.649133 31498 hybrid_clock.cc:648] HybridClock initialized: now 1790280791649116 us; error 35 us; skew 500 ppm
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:11.652412 31498 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:11.653448 31498 webserver.cc:460] Webserver started at http://127.29.83.131:43621/ using document root <none> and password file <none>
I20260924 20:13:11.654090 31498 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:11.654158 31498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:11.654500 31498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:11.656883 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "aabca806b3f82abff968952b534db9b6"
server_key_iv: "53d9d6370ed2ce416dc9f1e157084e4b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.657469 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "aabca806b3f82abff968952b534db9b6"
server_key_iv: "53d9d6370ed2ce416dc9f1e157084e4b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.661448 31498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:13:11.664077 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.665500 31498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:11.665632 31498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b"
format_stamp: "Formatted at 2026-09-24 20:13:11 on dist-test-slave-52fr"
server_key: "aabca806b3f82abff968952b534db9b6"
server_key_iv: "53d9d6370ed2ce416dc9f1e157084e4b"
server_key_version: "encryptionkey@0"
I20260924 20:13:11.665810 31498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:11.677242 31498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:11.680668 31498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:11.680884 31498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:11.682508 31498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:11.682559 31498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:11.682621 31498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:11.682651 31498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:11.694525 31498 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:35295
I20260924 20:13:11.694587 31628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:35295 every 8 connection(s)
I20260924 20:13:11.695636 31498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:11.696023 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31498
I20260924 20:13:11.696141 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:11.696440 30030 external_mini_cluster.cc:1428] Setting key 8096822c99d20095d342bf017967939c
I20260924 20:13:11.698104 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.683933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.131:37445)
0924 20:13:11.684368 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.684417 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.685494 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.686216 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.686227 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.686574 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.687107 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.687120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.687984 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.687990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.689156 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.689165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.689823 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.689828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.689987 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.690597 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.690621 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.692915 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.694992 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.694997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.695009 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.695268 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.695567 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.695569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.695571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.695741 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.696306 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.696312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.696592 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.696818 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.697094 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260924 20:13:11.698179 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.683531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37445 (local address 127.29.83.190:37595)
0924 20:13:11.683693 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.683699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.684518 (+ 819us) server_negotiation.cc:407] Connection header received
0924 20:13:11.685800 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.685804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.685876 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.685972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.687258 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.687810 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.689275 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.689605 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.689736 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.693084 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.693087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.693090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.693124 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.694864 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.695376 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.695379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.695381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.695473 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.695875 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.695880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.695885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.696135 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.696251 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.697904 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.698026 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:11.699280 31629 heartbeater.cc:347] Connected to a master server at 127.29.83.190:37595
I20260924 20:13:11.699411 31629 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:11.699934 31629 heartbeater.cc:511] Master 127.29.83.190:37595 requested a full tablet report, sending...
I20260924 20:13:11.701109 31145 ts_manager.cc:194] Registered new tserver with Master: e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131:35295)
I20260924 20:13:11.701915 31145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:37445
I20260924 20:13:11.709197 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.701571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:42881 (local address 127.29.83.190:37595)
0924 20:13:11.701690 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.701694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.701947 (+ 253us) server_negotiation.cc:407] Connection header received
0924 20:13:11.702113 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.702115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.702181 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.702292 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.702758 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.703513 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.704420 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.704637 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.704756 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.706076 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.706078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.706081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.706104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.707560 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.707998 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.708001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.708002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.708080 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.708377 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.708381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.708383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.708623 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.708711 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.708967 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.709064 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:13:11.709539 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.701453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.29.83.131:42881)
0924 20:13:11.701864 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.701878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.701980 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.702326 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.702329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.702549 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:11.702649 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.702655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.703657 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.703661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.704230 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.704237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.704794 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.704797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.704882 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.705491 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:11.705514 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:11.705938 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:11.707706 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.707708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.707710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.707906 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.708181 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:11.708183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:11.708184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:11.708258 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:11.708736 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:11.708739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:11.708827 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.708911 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.709192 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":104,"threads_started":1}
I20260924 20:13:11.710879 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:13:11 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:11.737947 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.729360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.29.83.190:37595)
0924 20:13:11.729506 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.729510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.729524 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:11.729743 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.729746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.729864 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.729949 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:11.730688 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.731224 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.732066 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.732289 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.732380 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.734740 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:11.734743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:11.734746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:11.734771 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:11.736218 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.736625 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.736628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.736629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.736729 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:11.736981 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:11.736984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:11.736985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:11.737179 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:11.737298 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.737546 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.737647 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:13:11.740944 31145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:11.744113 31145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:11.758651 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.752494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35295 (local address 127.0.0.1:40936)
0924 20:13:11.753551 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.753608 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.753782 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.754392 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.754398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.754420 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:11.754567 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.754574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.755834 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.755840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.757043 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.757052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.758037 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.758161 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.758207 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.758342 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.758471 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":93,"threads_started":1}
I20260924 20:13:11.758653 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.752599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.29.83.131:35295)
0924 20:13:11.753889 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.753904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.753924 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:13:11.754000 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.754007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.754232 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.754404 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:11.754698 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.755685 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.757162 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.757902 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758006 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.758174 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.758379 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.758496 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":116,"threads_started":1}
I20260924 20:13:11.759316 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.752829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:33549 (local address 127.0.0.1:53120)
0924 20:13:11.754002 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.754220 (+ 218us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.754315 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.754959 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.754962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.754982 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:11.755070 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.755076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.756468 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.756471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.757585 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.757591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.758858 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.759032 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.759057 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.759125 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.759192 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":43,"thread_start_us":79,"threads_started":1}
I20260924 20:13:11.759657 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.753363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.29.83.129:46563)
0924 20:13:11.753750 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.753797 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.753915 (+ 118us) server_negotiation.cc:407] Connection header received
0924 20:13:11.754171 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.754180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.754386 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.754536 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:11.754792 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.756007 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.757538 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.758398 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758499 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.758626 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.759222 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.759331 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":87,"threads_started":1}
I20260924 20:13:11.759773 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.753330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.29.83.130:33549)
0924 20:13:11.753996 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.754011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.754408 (+ 397us) server_negotiation.cc:407] Connection header received
0924 20:13:11.754492 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.754497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.754660 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.754814 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:11.755193 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.756346 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.757691 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.758527 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.758780 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.759533 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.759628 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":108,"threads_started":1}
I20260924 20:13:11.760236 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.752540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:46563 (local address 127.0.0.1:46992)
0924 20:13:11.753815 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.753829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.753990 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.754562 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.754564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.754578 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:11.754681 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.754687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.756142 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.756145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.757411 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.757421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.758695 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.758698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.759049 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.759074 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.760065 (+ 991us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.760145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":68,"thread_start_us":101,"threads_started":1}
I20260924 20:13:11.761665 31563 tablet_service.cc:1467] Processing CreateTablet for tablet c369279607944b9a9aae38601e73b6fc (DEFAULT_TABLE table=test-table [id=fc7624071c03472495ab601eeced6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:11.762072 31423 tablet_service.cc:1467] Processing CreateTablet for tablet c369279607944b9a9aae38601e73b6fc (DEFAULT_TABLE table=test-table [id=fc7624071c03472495ab601eeced6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:11.762842 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c369279607944b9a9aae38601e73b6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:11.763051 31423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c369279607944b9a9aae38601e73b6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:11.764214 31283 tablet_service.cc:1467] Processing CreateTablet for tablet c369279607944b9a9aae38601e73b6fc (DEFAULT_TABLE table=test-table [id=fc7624071c03472495ab601eeced6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:11.765153 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c369279607944b9a9aae38601e73b6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:11.770857 31653 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Bootstrap starting.
I20260924 20:13:11.771668 31652 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Bootstrap starting.
I20260924 20:13:11.772135 31654 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Bootstrap starting.
I20260924 20:13:11.773207 31653 tablet_bootstrap.cc:654] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:11.773625 31652 tablet_bootstrap.cc:654] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:11.774272 31654 tablet_bootstrap.cc:654] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:11.774542 31652 log.cc:824] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:11.774533 31653 log.cc:824] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:11.775050 31654 log.cc:824] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:11.776510 31652 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: No bootstrap required, opened a new log
I20260924 20:13:11.776741 31652 ts_tablet_manager.cc:1397] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:13:11.776911 31654 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: No bootstrap required, opened a new log
I20260924 20:13:11.776912 31653 tablet_bootstrap.cc:492] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: No bootstrap required, opened a new log
I20260924 20:13:11.777101 31653 ts_tablet_manager.cc:1397] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 20:13:11.777101 31654 ts_tablet_manager.cc:1397] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:13:11.779978 31652 raft_consensus.cc:348] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.780228 31652 raft_consensus.cc:374] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:11.780289 31652 raft_consensus.cc:729] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a2104ff87e4b03a7c42cf024aa9c2b, State: Initialized, Role: FOLLOWER
I20260924 20:13:11.780876 31652 consensus_queue.cc:260] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.781117 31653 raft_consensus.cc:348] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.781069 31654 raft_consensus.cc:348] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.781319 31653 raft_consensus.cc:374] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:11.781343 31654 raft_consensus.cc:374] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:11.781374 31653 raft_consensus.cc:729] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a30838ad913433dbff428455be70825, State: Initialized, Role: FOLLOWER
I20260924 20:13:11.781389 31654 raft_consensus.cc:729] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3569de8895b471681fd115a1e980a7e, State: Initialized, Role: FOLLOWER
I20260924 20:13:11.781875 31653 consensus_queue.cc:260] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.781875 31654 consensus_queue.cc:260] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.781983 31652 ts_tablet_manager.cc:1428] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:13:11.782101 31629 heartbeater.cc:503] Master 127.29.83.190:37595 was elected leader, sending a full tablet report...
I20260924 20:13:11.782487 31349 heartbeater.cc:503] Master 127.29.83.190:37595 was elected leader, sending a full tablet report...
I20260924 20:13:11.782841 31653 ts_tablet_manager.cc:1428] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:13:11.782841 31654 ts_tablet_manager.cc:1428] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:13:11.783172 31489 heartbeater.cc:503] Master 127.29.83.190:37595 was elected leader, sending a full tablet report...
I20260924 20:13:11.928292 31658 raft_consensus.cc:482] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:11.928498 31658 raft_consensus.cc:504] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.929805 31658 leader_election.cc:288] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563), 6a30838ad913433dbff428455be70825 (127.29.83.130:33549)
I20260924 20:13:11.933494 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.930121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:44409 (local address 127.29.83.129:46563)
0924 20:13:11.930271 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.930275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.930405 (+ 130us) server_negotiation.cc:407] Connection header received
0924 20:13:11.930616 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.930619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.930658 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.930742 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:11.931014 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.931542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.932551 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.932946 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.933070 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.933130 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.933308 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.933373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:11.933508 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.930038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:46563 (local address 127.29.83.131:44409)
0924 20:13:11.930285 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.930301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.930405 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.930755 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.930757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.930773 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:11.930909 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.930914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.931650 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.931655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.932430 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.932438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.933058 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.933061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.933175 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.933200 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.933286 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.933358 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":2}
I20260924 20:13:11.933964 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.930419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:42067 (local address 127.29.83.130:33549)
0924 20:13:11.930541 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.930545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.930557 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:11.930706 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.930713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.930752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.930834 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:11.931114 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.931607 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.932986 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.933376 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.933501 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.933546 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.933741 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.933843 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:13:11.934250 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c369279607944b9a9aae38601e73b6fc" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3569de8895b471681fd115a1e980a7e" is_pre_election: true
I20260924 20:13:11.934345 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.930101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:33549 (local address 127.29.83.131:42067)
0924 20:13:11.930480 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:11.930496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:11.930583 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:11.930845 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:11.930847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:11.930864 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:11.931008 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.931013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.931727 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.931733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.932861 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:11.932869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.933489 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.933492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:11.933594 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.933620 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:13:11.934115 (+ 495us) client_negotiation.cc:240] Negotiation successful
0924 20:13:11.934186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":21,"thread_start_us":148,"threads_started":1}
I20260924 20:13:11.934603 31303 raft_consensus.cc:2454] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 0.
I20260924 20:13:11.935104 31517 leader_election.cc:302] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0a2104ff87e4b03a7c42cf024aa9c2b, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:11.935026 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c369279607944b9a9aae38601e73b6fc" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825" is_pre_election: true
I20260924 20:13:11.935393 31443 raft_consensus.cc:2454] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 0.
I20260924 20:13:11.935369 31658 raft_consensus.cc:2781] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:11.935475 31658 raft_consensus.cc:482] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:11.935519 31658 raft_consensus.cc:3037] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:11.936674 31658 raft_consensus.cc:504] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.936997 31658 leader_election.cc:288] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 election: Requested vote from peers f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563), 6a30838ad913433dbff428455be70825 (127.29.83.130:33549)
I20260924 20:13:11.937341 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c369279607944b9a9aae38601e73b6fc" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825"
I20260924 20:13:11.937366 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c369279607944b9a9aae38601e73b6fc" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3569de8895b471681fd115a1e980a7e"
I20260924 20:13:11.937444 31443 raft_consensus.cc:3037] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:11.937502 31303 raft_consensus.cc:3037] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:11.938679 31443 raft_consensus.cc:2454] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 1.
I20260924 20:13:11.938827 31303 raft_consensus.cc:2454] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 1.
I20260924 20:13:11.939111 31517 leader_election.cc:302] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0a2104ff87e4b03a7c42cf024aa9c2b, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:11.939342 31658 raft_consensus.cc:2781] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:11.939559 31658 raft_consensus.cc:686] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 LEADER]: Becoming Leader. State: Replica: e0a2104ff87e4b03a7c42cf024aa9c2b, State: Running, Role: LEADER
I20260924 20:13:11.939890 31658 consensus_queue.cc:237] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } }
I20260924 20:13:11.943234 31145 catalog_manager.cc:5495] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131). New cstate: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } health_report { overall_health: UNKNOWN } } }
W20260924 20:13:11.946735 31630 tablet.cc:2369] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:11.977911 31490 tablet.cc:2369] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:11.999322 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:11.995144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.29.83.131:35295)
0924 20:13:11.995304 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:11.995308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:11.995322 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:11.995536 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:11.995539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:11.995595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:11.995684 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:11.996084 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.996727 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.997702 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:11.997999 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:11.998085 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:11.998387 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:11.998854 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:11.998962 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:11.999102 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 20:13:11.999158 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 20:13:12.002103 31350 tablet.cc:2369] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:12.005231 31443 raft_consensus.cc:1261] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Refusing update from remote peer e0a2104ff87e4b03a7c42cf024aa9c2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.005306 31303 raft_consensus.cc:1261] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Refusing update from remote peer e0a2104ff87e4b03a7c42cf024aa9c2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.005965 31658 consensus_queue.cc:1035] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.006165 31662 consensus_queue.cc:1035] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.026867 31145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:12.027089 31145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:12.029896 31145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Sending DeleteTablet for 3 replicas of tablet c369279607944b9a9aae38601e73b6fc
I20260924 20:13:12.030706 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet c369279607944b9a9aae38601e73b6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:40936
I20260924 20:13:12.030849 31423 tablet_service.cc:1514] Processing DeleteTablet for tablet c369279607944b9a9aae38601e73b6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53120
I20260924 20:13:12.030821 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet c369279607944b9a9aae38601e73b6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260924 20:13:12.031260 31674 tablet_replica.cc:330] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: stopping tablet replica
I20260924 20:13:12.031278 31675 tablet_replica.cc:330] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: stopping tablet replica
I20260924 20:13:12.031589 31674 raft_consensus.cc:2229] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:12.031760 31675 raft_consensus.cc:2229] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:12.032032 31674 raft_consensus.cc:2258] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.032189 31675 raft_consensus.cc:2258] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.032189 31145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:13:12.032462 31676 tablet_replica.cc:330] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: stopping tablet replica
W20260924 20:13:12.032651 31145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:12.032744 31676 raft_consensus.cc:2229] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:12.033054 31676 raft_consensus.cc:2258] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.034619 31675 ts_tablet_manager.cc:1905] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.034619 31674 ts_tablet_manager.cc:1905] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.037307 31675 ts_tablet_manager.cc:1918] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:12.037307 31674 ts_tablet_manager.cc:1918] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:12.037406 31675 log.cc:1196] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/c369279607944b9a9aae38601e73b6fc
I20260924 20:13:12.037406 31674 log.cc:1196] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/c369279607944b9a9aae38601e73b6fc
I20260924 20:13:12.037712 31675 ts_tablet_manager.cc:1939] T c369279607944b9a9aae38601e73b6fc P 6a30838ad913433dbff428455be70825: Deleting consensus metadata
I20260924 20:13:12.037712 31674 ts_tablet_manager.cc:1939] T c369279607944b9a9aae38601e73b6fc P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting consensus metadata
I20260924 20:13:12.038707 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 (DEFAULT_TABLE table=test-table [id=050d41723abd4d1eb1632eafeee05b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:12.038952 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3022536e0cc240178d5d23b8fb5d4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.039187 31423 tablet_service.cc:1467] Processing CreateTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 (DEFAULT_TABLE table=test-table [id=050d41723abd4d1eb1632eafeee05b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:12.039220 31283 tablet_service.cc:1467] Processing CreateTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 (DEFAULT_TABLE table=test-table [id=050d41723abd4d1eb1632eafeee05b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:12.039413 31423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3022536e0cc240178d5d23b8fb5d4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.039480 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3022536e0cc240178d5d23b8fb5d4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.041417 31130 catalog_manager.cc:4841] TS e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131:35295): tablet c369279607944b9a9aae38601e73b6fc (table test-table [id=fc7624071c03472495ab601eeced6309]) successfully deleted
I20260924 20:13:12.041782 31652 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Bootstrap starting.
I20260924 20:13:12.041841 31654 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Bootstrap starting.
I20260924 20:13:12.042753 31652 tablet_bootstrap.cc:654] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.042827 31676 ts_tablet_manager.cc:1905] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.042893 31654 tablet_bootstrap.cc:654] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.043402 31653 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Bootstrap starting.
I20260924 20:13:12.044189 31654 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: No bootstrap required, opened a new log
I20260924 20:13:12.044270 31654 ts_tablet_manager.cc:1397] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:12.044385 31653 tablet_bootstrap.cc:654] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.044704 31654 raft_consensus.cc:348] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.044808 31654 raft_consensus.cc:374] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.044838 31654 raft_consensus.cc:729] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3569de8895b471681fd115a1e980a7e, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.044976 31654 consensus_queue.cc:260] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.045620 31676 ts_tablet_manager.cc:1918] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:12.045719 31676 log.cc:1196] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/c369279607944b9a9aae38601e73b6fc
I20260924 20:13:12.046468 31676 ts_tablet_manager.cc:1939] T c369279607944b9a9aae38601e73b6fc P f3569de8895b471681fd115a1e980a7e: Deleting consensus metadata
I20260924 20:13:12.046870 31653 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: No bootstrap required, opened a new log
I20260924 20:13:12.046947 31653 ts_tablet_manager.cc:1397] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 20:13:12.047372 31653 raft_consensus.cc:348] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.047487 31653 raft_consensus.cc:374] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.047516 31653 raft_consensus.cc:729] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a30838ad913433dbff428455be70825, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.047642 31653 consensus_queue.cc:260] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.047821 31133 catalog_manager.cc:4841] TS f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563): tablet c369279607944b9a9aae38601e73b6fc (table test-table [id=fc7624071c03472495ab601eeced6309]) successfully deleted
I20260924 20:13:12.047941 31653 ts_tablet_manager.cc:1428] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:12.048170 31654 ts_tablet_manager.cc:1428] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260924 20:13:12.048781 31132 catalog_manager.cc:4841] TS 6a30838ad913433dbff428455be70825 (127.29.83.130:33549): tablet c369279607944b9a9aae38601e73b6fc (table test-table [id=fc7624071c03472495ab601eeced6309]) successfully deleted
I20260924 20:13:12.050262 31652 tablet_bootstrap.cc:492] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: No bootstrap required, opened a new log
I20260924 20:13:12.050371 31652 ts_tablet_manager.cc:1397] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 20:13:12.050812 31652 raft_consensus.cc:348] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.050915 31652 raft_consensus.cc:374] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.050936 31652 raft_consensus.cc:729] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a2104ff87e4b03a7c42cf024aa9c2b, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.051085 31652 consensus_queue.cc:260] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.051352 31652 ts_tablet_manager.cc:1428] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:13:12.227313 31659 raft_consensus.cc:482] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:12.227470 31659 raft_consensus.cc:504] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.228626 31659 leader_election.cc:288] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a30838ad913433dbff428455be70825 (127.29.83.130:33549), e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131:35295)
I20260924 20:13:12.232548 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.229169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:59845 (local address 127.29.83.131:35295)
0924 20:13:12.229298 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.229303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.229440 (+ 137us) server_negotiation.cc:407] Connection header received
0924 20:13:12.229597 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.229600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.229646 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.229736 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:12.230008 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.230496 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.231530 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.231939 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.232067 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.232126 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.232362 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.232425 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:13:12.232548 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.228948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:44677 (local address 127.29.83.130:33549)
0924 20:13:12.229084 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.229092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.229326 (+ 234us) server_negotiation.cc:407] Connection header received
0924 20:13:12.229499 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.229503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.229548 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.229634 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:12.229948 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.230444 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.231526 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.231918 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.232051 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.232106 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.232348 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.232415 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:13:12.232579 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.228815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:33549 (local address 127.29.83.129:44677)
0924 20:13:12.229222 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.229243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.229397 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.229660 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.229663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.229678 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:12.229842 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.229853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.230559 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.230562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.231407 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.231417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.232055 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.232059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.232203 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.232248 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.232333 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.232409 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":33,"thread_start_us":93,"threads_started":1}
I20260924 20:13:12.232586 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.228964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35295 (local address 127.29.83.129:59845)
0924 20:13:12.229350 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.229361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.229492 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.229745 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.229748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.229785 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:12.229905 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.229910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.230606 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.230611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.231411 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.231420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.232060 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.232063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.232211 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.232246 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.232344 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.232416 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":36,"thread_start_us":109,"threads_started":1}
I20260924 20:13:12.233070 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3022536e0cc240178d5d23b8fb5d4023" candidate_uuid: "f3569de8895b471681fd115a1e980a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825" is_pre_election: true
I20260924 20:13:12.233224 31443 raft_consensus.cc:2454] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3569de8895b471681fd115a1e980a7e in term 0.
I20260924 20:13:12.233520 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3022536e0cc240178d5d23b8fb5d4023" candidate_uuid: "f3569de8895b471681fd115a1e980a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" is_pre_election: true
I20260924 20:13:12.233575 31236 leader_election.cc:302] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a30838ad913433dbff428455be70825, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:12.233801 31583 raft_consensus.cc:2454] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3569de8895b471681fd115a1e980a7e in term 0.
I20260924 20:13:12.233898 31659 raft_consensus.cc:2781] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:12.233958 31659 raft_consensus.cc:482] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:12.234014 31659 raft_consensus.cc:3037] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.234997 31659 raft_consensus.cc:504] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.235345 31659 leader_election.cc:288] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [CANDIDATE]: Term 1 election: Requested vote from peers 6a30838ad913433dbff428455be70825 (127.29.83.130:33549), e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131:35295)
I20260924 20:13:12.235677 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3022536e0cc240178d5d23b8fb5d4023" candidate_uuid: "f3569de8895b471681fd115a1e980a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b"
I20260924 20:13:12.235793 31583 raft_consensus.cc:3037] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.235854 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3022536e0cc240178d5d23b8fb5d4023" candidate_uuid: "f3569de8895b471681fd115a1e980a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825"
I20260924 20:13:12.236001 31443 raft_consensus.cc:3037] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.236763 31583 raft_consensus.cc:2454] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3569de8895b471681fd115a1e980a7e in term 1.
I20260924 20:13:12.236967 31443 raft_consensus.cc:2454] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3569de8895b471681fd115a1e980a7e in term 1.
I20260924 20:13:12.237048 31234 leader_election.cc:302] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0a2104ff87e4b03a7c42cf024aa9c2b, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:12.237351 31659 raft_consensus.cc:2781] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:12.237529 31659 raft_consensus.cc:686] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 1 LEADER]: Becoming Leader. State: Replica: f3569de8895b471681fd115a1e980a7e, State: Running, Role: LEADER
I20260924 20:13:12.237838 31659 consensus_queue.cc:237] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.240186 31144 catalog_manager.cc:5495] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to f3569de8895b471681fd115a1e980a7e (127.29.83.129). New cstate: current_term: 1 leader_uuid: "f3569de8895b471681fd115a1e980a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:13:12 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280790, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:12.316713 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.307677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.29.83.190:37595)
0924 20:13:12.308014 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.308024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.308141 (+ 117us) server_negotiation.cc:407] Connection header received
0924 20:13:12.308311 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.308315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.308357 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.308448 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:12.309119 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.309678 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.310496 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.310710 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.310818 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.313295 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:12.313299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:12.313301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:12.313322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:12.314878 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.315331 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.315337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.315339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.315442 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.315770 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.315774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.315776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.315968 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:12.316068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.316363 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.316568 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":263,"thread_start_us":168,"threads_started":1}
I20260924 20:13:12.316713 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.307676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:37595 (local address 127.0.0.1:35808)
0924 20:13:12.308043 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.308058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.308201 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.308463 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.308466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.308904 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:12.308995 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.309001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.309810 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.309815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.310379 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.310386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.310823 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.310826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.310913 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.311496 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:12.311508 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:12.313133 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:12.314988 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:12.315001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:12.315008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:12.315235 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:12.315566 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:12.315569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:12.315574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:12.315674 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:12.316068 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:12.316077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:12.316188 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.316318 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.316508 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":4,"thread_start_us":150,"threads_started":1}
I20260924 20:13:12.320827 31144 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:35808:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:13:12.321483 31144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:12.326177 31283 tablet_service.cc:1467] Processing CreateTablet for tablet 2db455f971134e70acbaad984d483495 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c30af41c5eeb41889961e04ca6a8fc6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:12.326380 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 2db455f971134e70acbaad984d483495 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c30af41c5eeb41889961e04ca6a8fc6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:12.326364 31423 tablet_service.cc:1467] Processing CreateTablet for tablet 2db455f971134e70acbaad984d483495 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c30af41c5eeb41889961e04ca6a8fc6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:12.326591 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db455f971134e70acbaad984d483495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.326709 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db455f971134e70acbaad984d483495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.326800 31423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db455f971134e70acbaad984d483495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:12.329349 31653 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: Bootstrap starting.
I20260924 20:13:12.329386 31652 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: Bootstrap starting.
I20260924 20:13:12.330302 31652 tablet_bootstrap.cc:654] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.330308 31653 tablet_bootstrap.cc:654] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.330853 31654 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: Bootstrap starting.
I20260924 20:13:12.331461 31652 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: No bootstrap required, opened a new log
I20260924 20:13:12.331550 31652 ts_tablet_manager.cc:1397] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 20:13:12.331692 31654 tablet_bootstrap.cc:654] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:12.332404 31652 raft_consensus.cc:348] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.332533 31652 raft_consensus.cc:374] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.332567 31652 raft_consensus.cc:729] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a2104ff87e4b03a7c42cf024aa9c2b, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.333096 31653 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: No bootstrap required, opened a new log
I20260924 20:13:12.333189 31653 ts_tablet_manager.cc:1397] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 20:13:12.333571 31653 raft_consensus.cc:348] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.333642 31654 tablet_bootstrap.cc:492] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: No bootstrap required, opened a new log
I20260924 20:13:12.333673 31653 raft_consensus.cc:374] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.333695 31653 raft_consensus.cc:729] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a30838ad913433dbff428455be70825, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.333737 31654 ts_tablet_manager.cc:1397] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:12.333863 31653 consensus_queue.cc:260] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.334136 31653 ts_tablet_manager.cc:1428] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:13:12.334092 31660 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.334230 31660 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.334221 31654 raft_consensus.cc:348] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.334368 31654 raft_consensus.cc:374] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:12.334404 31654 raft_consensus.cc:729] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3569de8895b471681fd115a1e980a7e, State: Initialized, Role: FOLLOWER
I20260924 20:13:12.334532 31654 consensus_queue.cc:260] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.334808 31654 ts_tablet_manager.cc:1428] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:12.335001 31659 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.335127 31659 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.334697 31652 consensus_queue.cc:260] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.335719 31658 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.335803 31652 ts_tablet_manager.cc:1428] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 20:13:12.335842 31658 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.340942 31658 raft_consensus.cc:482] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:12.341042 31658 raft_consensus.cc:504] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.341439 31658 leader_election.cc:288] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563), 6a30838ad913433dbff428455be70825 (127.29.83.130:33549)
I20260924 20:13:12.341730 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db455f971134e70acbaad984d483495" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3569de8895b471681fd115a1e980a7e" is_pre_election: true
I20260924 20:13:12.341794 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db455f971134e70acbaad984d483495" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825" is_pre_election: true
I20260924 20:13:12.341878 31303 raft_consensus.cc:2454] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 0.
I20260924 20:13:12.341892 31443 raft_consensus.cc:2454] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 0.
I20260924 20:13:12.342124 31517 leader_election.cc:302] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0a2104ff87e4b03a7c42cf024aa9c2b, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:12.342326 31658 raft_consensus.cc:2781] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:12.342365 31658 raft_consensus.cc:482] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:12.342389 31658 raft_consensus.cc:3037] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.343223 31658 raft_consensus.cc:504] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.343494 31658 leader_election.cc:288] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 election: Requested vote from peers f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563), 6a30838ad913433dbff428455be70825 (127.29.83.130:33549)
I20260924 20:13:12.343796 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db455f971134e70acbaad984d483495" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3569de8895b471681fd115a1e980a7e"
I20260924 20:13:12.343801 31443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db455f971134e70acbaad984d483495" candidate_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a30838ad913433dbff428455be70825"
I20260924 20:13:12.343883 31443 raft_consensus.cc:3037] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.343910 31303 raft_consensus.cc:3037] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:12.345083 31303 raft_consensus.cc:2454] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 1.
I20260924 20:13:12.345094 31443 raft_consensus.cc:2454] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2104ff87e4b03a7c42cf024aa9c2b in term 1.
I20260924 20:13:12.345356 31517 leader_election.cc:302] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0a2104ff87e4b03a7c42cf024aa9c2b, f3569de8895b471681fd115a1e980a7e; no voters:
I20260924 20:13:12.345573 31658 raft_consensus.cc:2781] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:12.345656 31658 raft_consensus.cc:686] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 LEADER]: Becoming Leader. State: Replica: e0a2104ff87e4b03a7c42cf024aa9c2b, State: Running, Role: LEADER
I20260924 20:13:12.345795 31658 consensus_queue.cc:237] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } }
I20260924 20:13:12.346443 31662 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: TxnStatusTablet state changed. Reason: New leader e0a2104ff87e4b03a7c42cf024aa9c2b. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.346546 31662 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:12.346774 31695 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:13:12.346916 31695 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:13:12.347296 31144 catalog_manager.cc:5495] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131). New cstate: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } health_report { overall_health: HEALTHY } } }
I20260924 20:13:12.361040 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.357842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.29.83.131:35295)
0924 20:13:12.357952 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.357956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.357968 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:12.358036 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.358038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.358077 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.358138 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:12.358395 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.358882 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.360055 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.360269 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.360347 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.360647 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.360710 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.360763 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.360889 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.360934 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:13:12.361097 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.357602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35295 (local address 127.0.0.1:40942)
0924 20:13:12.357810 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.357828 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.357934 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.358176 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.358179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.358187 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:12.358294 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.358301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.358975 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.358978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.359930 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.359941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.360370 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.360373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.360470 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.360525 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.360793 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.360807 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.360877 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.360959 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:13:12.363422 31443 raft_consensus.cc:1261] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Refusing update from remote peer e0a2104ff87e4b03a7c42cf024aa9c2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.363581 31303 raft_consensus.cc:1261] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Refusing update from remote peer e0a2104ff87e4b03a7c42cf024aa9c2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.363837 31662 consensus_queue.cc:1035] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.363971 31658 consensus_queue.cc:1035] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.365317 31660 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: TxnStatusTablet state changed. Reason: New leader e0a2104ff87e4b03a7c42cf024aa9c2b. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.365422 31660 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.366008 31659 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: TxnStatusTablet state changed. Reason: New leader e0a2104ff87e4b03a7c42cf024aa9c2b. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.366110 31659 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.367349 31665 mvcc.cc:205] Tried to move back new op lower bound from 7332990125516795904 to 7332990125449580544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990125516795904}]
I20260924 20:13:12.367466 31662 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.367568 31662 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:12.367965 31666 mvcc.cc:205] Tried to move back new op lower bound from 7332990125516795904 to 7332990125449580544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990125516795904}]
I20260924 20:13:12.368108 31660 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.368227 31660 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P 6a30838ad913433dbff428455be70825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.369796 31658 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.369905 31658 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P e0a2104ff87e4b03a7c42cf024aa9c2b: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:12.370669 31667 mvcc.cc:205] Tried to move back new op lower bound from 7332990125516795904 to 7332990125449580544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990125516795904}]
I20260924 20:13:12.371351 31659 tablet_replica.cc:439] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3569de8895b471681fd115a1e980a7e" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 46563 } } peers { permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 } } peers { permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 } } }
I20260924 20:13:12.371457 31659 tablet_replica.cc:442] T 2db455f971134e70acbaad984d483495 P f3569de8895b471681fd115a1e980a7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:12.377322 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.373002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.29.83.129:46563)
0924 20:13:12.373121 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.373125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.373191 (+ 66us) server_negotiation.cc:407] Connection header received
0924 20:13:12.373347 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.373350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.373409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.373513 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:12.373897 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.374672 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.375582 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.375929 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.376040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.376310 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.376893 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.377013 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.377123 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.377183 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:13:12.389416 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.385289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35295 (local address 127.29.83.129:53411)
0924 20:13:12.385742 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.385796 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.385910 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.386208 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.386211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.386219 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:12.386343 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.386349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.387040 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.387043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.388057 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.388068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.388535 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.388539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.388636 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.388692 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.389045 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.389051 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.389117 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.389197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":126,"threads_started":1}
I20260924 20:13:12.389441 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.385429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:53411 (local address 127.29.83.131:35295)
0924 20:13:12.385561 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.385566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.385873 (+ 307us) server_negotiation.cc:407] Connection header received
0924 20:13:12.386034 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.386037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.386110 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.386186 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:12.386446 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.386942 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.388199 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.388425 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.388512 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.388865 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.388955 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.389030 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.389161 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.389282 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:13:12.396801 31443 raft_consensus.cc:1261] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Refusing update from remote peer f3569de8895b471681fd115a1e980a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.397099 31583 raft_consensus.cc:1261] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Refusing update from remote peer f3569de8895b471681fd115a1e980a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:12.397279 31682 consensus_queue.cc:1035] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a30838ad913433dbff428455be70825" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.397565 31659 consensus_queue.cc:1035] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a2104ff87e4b03a7c42cf024aa9c2b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:12.426090 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.418355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:47355 (local address 127.29.83.129:46563)
0924 20:13:12.418487 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.418492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.422143 (+ 3651us) server_negotiation.cc:407] Connection header received
0924 20:13:12.422313 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.422316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.422376 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.422458 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:12.422980 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.423660 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.424984 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.425206 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.425288 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.425612 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.425686 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.425744 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.425881 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.425930 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:13:12.426147 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.418234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:46563 (local address 127.29.83.131:47355)
0924 20:13:12.422041 (+ 3807us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:12.422057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:12.422172 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:12.422482 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:12.422486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:12.422494 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:12.422607 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.422612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.423866 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.423870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.424860 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:12.424871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.425312 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.425315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:12.425416 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.425476 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:12.425785 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:12.425792 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:13:12.425861 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:13:12.425945 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3669,"thread_start_us":113,"threads_started":1}
I20260924 20:13:12.450659 31145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:12.450839 31145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:12.453016 31145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3986d68d88524d81b9f1c7d9621fe004: Sending DeleteTablet for 3 replicas of tablet 3022536e0cc240178d5d23b8fb5d4023
I20260924 20:13:12.453548 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260924 20:13:12.453783 31676 tablet_replica.cc:330] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: stopping tablet replica
I20260924 20:13:12.453802 31423 tablet_service.cc:1514] Processing DeleteTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53120
I20260924 20:13:12.453913 31676 raft_consensus.cc:2229] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:12.454160 31676 raft_consensus.cc:2258] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.454360 31675 tablet_replica.cc:330] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: stopping tablet replica
I20260924 20:13:12.454481 31675 raft_consensus.cc:2229] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:12.454603 31675 raft_consensus.cc:2258] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.455323 31675 ts_tablet_manager.cc:1905] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.455848 31676 ts_tablet_manager.cc:1905] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.458217 31675 ts_tablet_manager.cc:1918] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:12.458287 31675 log.cc:1196] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/3022536e0cc240178d5d23b8fb5d4023
I20260924 20:13:12.458539 31675 ts_tablet_manager.cc:1939] T 3022536e0cc240178d5d23b8fb5d4023 P 6a30838ad913433dbff428455be70825: Deleting consensus metadata
I20260924 20:13:12.458660 31676 ts_tablet_manager.cc:1918] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:12.458737 31676 log.cc:1196] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/3022536e0cc240178d5d23b8fb5d4023
I20260924 20:13:12.459003 31676 ts_tablet_manager.cc:1939] T 3022536e0cc240178d5d23b8fb5d4023 P f3569de8895b471681fd115a1e980a7e: Deleting consensus metadata
I20260924 20:13:12.459242 31132 catalog_manager.cc:4841] TS 6a30838ad913433dbff428455be70825 (127.29.83.130:33549): tablet 3022536e0cc240178d5d23b8fb5d4023 (table test-table [id=050d41723abd4d1eb1632eafeee05b1b]) successfully deleted
I20260924 20:13:12.459694 31133 catalog_manager.cc:4841] TS f3569de8895b471681fd115a1e980a7e (127.29.83.129:46563): tablet 3022536e0cc240178d5d23b8fb5d4023 (table test-table [id=050d41723abd4d1eb1632eafeee05b1b]) successfully deleted
I20260924 20:13:12.460096 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 3022536e0cc240178d5d23b8fb5d4023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:40936
I20260924 20:13:12.460300 31674 tablet_replica.cc:330] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: stopping tablet replica
I20260924 20:13:12.460413 31674 raft_consensus.cc:2229] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:12.460515 31674 raft_consensus.cc:2258] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:12.461205 31674 ts_tablet_manager.cc:1905] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:12.464043 31674 ts_tablet_manager.cc:1918] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:12.464116 31674 log.cc:1196] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/3022536e0cc240178d5d23b8fb5d4023
I20260924 20:13:12.464358 31674 ts_tablet_manager.cc:1939] T 3022536e0cc240178d5d23b8fb5d4023 P e0a2104ff87e4b03a7c42cf024aa9c2b: Deleting consensus metadata
I20260924 20:13:12.465010 31130 catalog_manager.cc:4841] TS e0a2104ff87e4b03a7c42cf024aa9c2b (127.29.83.131:35295): tablet 3022536e0cc240178d5d23b8fb5d4023 (table test-table [id=050d41723abd4d1eb1632eafeee05b1b]) successfully deleted
Sep 24 20:13:12 dist-test-slave-52fr krb5kdc[31101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:12.474165 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.460659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.29.83.129:46563)
0924 20:13:12.460782 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.460787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.461777 (+ 990us) server_negotiation.cc:407] Connection header received
0924 20:13:12.462003 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.462007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.462056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.462124 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:12.463183 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.463957 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.465215 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.465555 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.465635 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.469880 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:12.469891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:12.469908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:12.469937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:12.472087 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.472625 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.472632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.472634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.472732 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.473043 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.473046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.473047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.473362 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:12.473577 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.473854 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.473983 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 20:13:12.474617 31343 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47006
I20260924 20:13:12.492682 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:12.478259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.29.83.190:37595)
0924 20:13:12.478416 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:12.478421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:12.482092 (+ 3671us) server_negotiation.cc:407] Connection header received
0924 20:13:12.482366 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:12.482370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:12.482429 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:12.482491 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:12.483618 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.484460 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.485678 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:12.486092 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:12.486181 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:12.488207 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:12.488212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:12.488215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:12.488250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:12.490516 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.491106 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.491110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.491112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.491213 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:12.491545 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:12.491549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:12.491552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:12.491801 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:12.491917 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:12.492330 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 20:13:12.492487 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 20:13:12.494513 31145 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35824
I20260924 20:13:12.496914 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31218
I20260924 20:13:12.504511 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31358
I20260924 20:13:12.511211 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31498
I20260924 20:13:12.518342 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31114
2026-09-24T20:13:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2943 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:12 dist-test-slave-52fr krb5kdc[31716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:12 dist-test-slave-52fr krb5kdc[31716](info): set up 2 sockets
Sep 24 20:13:12 dist-test-slave-52fr krb5kdc[31716](info): commencing operation
krb5kdc: starting...
W20260924 20:13:13.561570 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:13 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:13.688447 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40875
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:13.799160 31729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:13.799477 31729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:13.799551 31729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:13.803608 31729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:13.803692 31729 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:13:13.803728 31729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:13.803771 31729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:13.803812 31729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:13.803846 31729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:13.809172 31729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:13.810367 31729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:13.811419 31729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:13.820182 31734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:13.820266 31735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:13.820160 31737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:13.820462 31729 server_base.cc:1034] running on GCE node
I20260924 20:13:13.821105 31729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:13.822359 31729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:13.823549 31729 hybrid_clock.cc:648] HybridClock initialized: now 1790280793823534 us; error 32 us; skew 500 ppm
Sep 24 20:13:13 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280793, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:13.826395 31729 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:13.827363 31729 webserver.cc:460] Webserver started at http://127.29.83.190:38041/ using document root <none> and password file <none>
I20260924 20:13:13.827934 31729 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:13.828024 31729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:13.828251 31729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:13.830411 31729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17"
format_stamp: "Formatted at 2026-09-24 20:13:13 on dist-test-slave-52fr"
server_key: "2b0b791761650ec8bff2e4eadf74452b"
server_key_iv: "bf20aea894ca9030cef9131c0ff890ce"
server_key_version: "encryptionkey@0"
I20260924 20:13:13.830989 31729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17"
format_stamp: "Formatted at 2026-09-24 20:13:13 on dist-test-slave-52fr"
server_key: "2b0b791761650ec8bff2e4eadf74452b"
server_key_iv: "bf20aea894ca9030cef9131c0ff890ce"
server_key_version: "encryptionkey@0"
I20260924 20:13:13.834745 31729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:13:13.837153 31744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:13.838398 31729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:13:13.838541 31729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17"
format_stamp: "Formatted at 2026-09-24 20:13:13 on dist-test-slave-52fr"
server_key: "2b0b791761650ec8bff2e4eadf74452b"
server_key_iv: "bf20aea894ca9030cef9131c0ff890ce"
server_key_version: "encryptionkey@0"
I20260924 20:13:13.838654 31729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:13.852559 31729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:13.856218 31729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:13.856388 31729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:13.864235 31729 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40875
I20260924 20:13:13.864279 31806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40875 every 8 connection(s)
I20260924 20:13:13.865208 31729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:13.868005 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:13.873483 31807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Bootstrap starting.
I20260924 20:13:13.874840 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31729
I20260924 20:13:13.874934 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:13.875197 30030 external_mini_cluster.cc:1428] Setting key 0121533d4b4f24e295d8cec0f55e6f01
I20260924 20:13:13.876106 31807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:13.876950 31807 log.cc:824] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:13.878924 31807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: No bootstrap required, opened a new log
I20260924 20:13:13.882570 31807 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } }
I20260924 20:13:13.882803 31807 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:13.882858 31807 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7035d8d5bb74dc98ff0e01d32ea3f17, State: Initialized, Role: FOLLOWER
I20260924 20:13:13.883284 31807 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } }
I20260924 20:13:13.883402 31807 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:13.883484 31807 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:13.883579 31807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 20:13:13 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:13.884812 31807 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } }
I20260924 20:13:13.885156 31807 leader_election.cc:302] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7035d8d5bb74dc98ff0e01d32ea3f17; no voters:
I20260924 20:13:13.885469 31807 leader_election.cc:288] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:13.885604 31812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:13.885852 31812 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [term 1 LEADER]: Becoming Leader. State: Replica: b7035d8d5bb74dc98ff0e01d32ea3f17, State: Running, Role: LEADER
I20260924 20:13:13.886296 31812 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } }
I20260924 20:13:13.886838 31807 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:13.888104 31813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } } }
I20260924 20:13:13.888234 31813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:13.888643 31814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7035d8d5bb74dc98ff0e01d32ea3f17. Latest consensus state: current_term: 1 leader_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7035d8d5bb74dc98ff0e01d32ea3f17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40875 } } }
I20260924 20:13:13.888796 31814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:13.889091 31822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:13.891273 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:13.876357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.29.83.190:40875)
0924 20:13:13.876820 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:13.876836 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:13.876872 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:13:13.877730 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:13.877816 (+ 86us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:13.878149 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:13.878491 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:13.879293 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:13.880379 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:13.881261 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:13.881896 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:13.882058 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:13.885018 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:13.885023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:13.885037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:13.885065 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:13.887302 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:13.887899 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:13.887905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:13.887912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:13.888070 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:13.888391 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:13.888397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:13.888400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:13.888999 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:13.889206 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:13.889977 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 20:13:13.890282 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260924 20:13:13.892625 31822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:13.898207 31822 catalog_manager.cc:1347] Generated new cluster ID: b6d2f1a1d139498fa1f00dfbdfa1cdb9
I20260924 20:13:13.898299 31822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:13.915683 31822 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:13.917043 31822 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:13.927700 31822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Generated new TSK 0
I20260924 20:13:13.928609 31822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:13.964828 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40875
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:14.076222 31833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:14.076506 31833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:14.076558 31833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:14.080886 31833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:14.080969 31833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:14.081014 31833 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:13:14.081028 31833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:14.081094 31833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:14.087244 31833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:14.088537 31833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:14.089660 31833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:14.099661 31838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:14.099704 31833 server_base.cc:1034] running on GCE node
W20260924 20:13:14.099661 31841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:14.100031 31839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:14.100494 31833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:14.101199 31833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:14.102413 31833 hybrid_clock.cc:648] HybridClock initialized: now 1790280794102392 us; error 35 us; skew 500 ppm
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:14.105427 31833 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:14.106603 31833 webserver.cc:460] Webserver started at http://127.29.83.129:33575/ using document root <none> and password file <none>
I20260924 20:13:14.107224 31833 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:14.107321 31833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:14.107561 31833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:14.109722 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "d42122042a264557bd70a015a84c173a"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "98ad36fba9014045ea59431404e92eb4"
server_key_iv: "6e96ee533f31b216036cd4745e6aeaa6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.110392 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "d42122042a264557bd70a015a84c173a"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "98ad36fba9014045ea59431404e92eb4"
server_key_iv: "6e96ee533f31b216036cd4745e6aeaa6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.114270 31833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:13:14.116758 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.118040 31833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:14.118167 31833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "d42122042a264557bd70a015a84c173a"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "98ad36fba9014045ea59431404e92eb4"
server_key_iv: "6e96ee533f31b216036cd4745e6aeaa6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.118278 31833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:14.134951 31833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:14.138401 31833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:14.138602 31833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:14.140089 31833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:14.140161 31833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.140229 31833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:14.140277 31833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:14.151434 31833 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:40295
I20260924 20:13:14.151482 31963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:40295 every 8 connection(s)
I20260924 20:13:14.152592 31833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:14.154662 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.141095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:56567 (local address 127.29.83.190:40875)
0924 20:13:14.141230 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.141233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.142443 (+ 1210us) server_negotiation.cc:407] Connection header received
0924 20:13:14.143614 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.143617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.143679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.143783 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.145514 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.146181 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.147381 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.147816 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.147949 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.150670 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.150674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.150677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.150712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.152584 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.153178 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.153182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.153184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.153275 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.153647 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.153650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.153651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.153900 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.153986 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.154425 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.154539 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:14.155539 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.141673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.129:56567)
0924 20:13:14.142279 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.142337 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.143394 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.144039 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.144053 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.144559 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.145331 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.145356 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.146315 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.146318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.147216 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.147246 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.148000 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.148006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.148237 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.148885 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.148902 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.150503 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.152714 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.152720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.152735 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.153050 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.153381 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.153385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.153387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.153545 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.153994 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.153999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.154222 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.154382 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.154627 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":136,"threads_started":1}
I20260924 20:13:14.156857 31964 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40875
I20260924 20:13:14.157013 31964 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:14.157483 31964 heartbeater.cc:511] Master 127.29.83.190:40875 requested a full tablet report, sending...
I20260924 20:13:14.159116 31761 ts_manager.cc:194] Registered new tserver with Master: d42122042a264557bd70a015a84c173a (127.29.83.129:40295)
I20260924 20:13:14.160475 31761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:56567
I20260924 20:13:14.161490 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31833
I20260924 20:13:14.161614 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:14.161955 30030 external_mini_cluster.cc:1428] Setting key b2871cd1832b6a6fc073693e2ec3049e
I20260924 20:13:14.166920 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.159206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:45669 (local address 127.29.83.190:40875)
0924 20:13:14.159342 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.159349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.159564 (+ 215us) server_negotiation.cc:407] Connection header received
0924 20:13:14.159647 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.159650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.159692 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.159784 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.160192 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.160894 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.162102 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.162386 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.162501 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.163653 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.163659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.163665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.163697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.165215 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.165713 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.165716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.165717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.165870 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.166159 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.166161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.166162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.166328 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.166403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.166688 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.166806 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:13:14.167102 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.159131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.129:45669)
0924 20:13:14.159480 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.159491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.159575 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.159810 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.159812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.159992 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.160090 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.160095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.161084 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.161091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.161951 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.161962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.162504 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.162508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.162600 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.163178 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.163195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.163520 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.165326 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.165331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.165335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.165611 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.165962 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.165965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.165967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.166072 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.166419 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.166425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.166540 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.166656 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.166822 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:14.189495 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40875
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:14.300081 31973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:14.300341 31973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:14.300390 31973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:14.304407 31973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:14.304505 31973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:14.304584 31973 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:13:14.304615 31973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:14.304649 31973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:14.310000 31973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:14.311028 31973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:14.312039 31973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:14.321600 31973 server_base.cc:1034] running on GCE node
W20260924 20:13:14.321641 31979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:14.321636 31978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:14.321806 31981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:14.322263 31973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:14.322930 31973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:14.324116 31973 hybrid_clock.cc:648] HybridClock initialized: now 1790280794324080 us; error 49 us; skew 500 ppm
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:14.326939 31973 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:14.327800 31973 webserver.cc:460] Webserver started at http://127.29.83.130:46339/ using document root <none> and password file <none>
I20260924 20:13:14.328356 31973 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:14.328444 31973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:14.328665 31973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:14.330785 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "044aa946d0f272f079ee161cdf8224f6"
server_key_iv: "c36342c39f048d997d6e8d6ddcec0201"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.331367 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "044aa946d0f272f079ee161cdf8224f6"
server_key_iv: "c36342c39f048d997d6e8d6ddcec0201"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.335304 31973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:14.337875 31988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.339123 31973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:14.339257 31973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "044aa946d0f272f079ee161cdf8224f6"
server_key_iv: "c36342c39f048d997d6e8d6ddcec0201"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.339381 31973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:14.373193 31973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:14.376283 31973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:14.376484 31973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:14.377997 31973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:14.378063 31973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.378130 31973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:14.378173 31973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:14.389024 31973 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:40951
I20260924 20:13:14.389278 32103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:40951 every 8 connection(s)
I20260924 20:13:14.390168 31973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:14.391984 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.378907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:38701 (local address 127.29.83.190:40875)
0924 20:13:14.379077 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.379085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.379915 (+ 830us) server_negotiation.cc:407] Connection header received
0924 20:13:14.380955 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.380961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.381033 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.381138 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.382583 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.383114 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.384448 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.384848 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.384983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.388060 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.388064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.388066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.388095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.389917 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.390464 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.390470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.390474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.390577 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.390968 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.390972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.390974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.391201 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.391287 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.391725 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.391851 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:14.392861 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.379298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.130:38701)
0924 20:13:14.379761 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.379810 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.380643 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.381386 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.381397 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.381831 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.382395 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.382420 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.383239 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.383241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.384340 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.384350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.385022 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.385026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.385244 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.385848 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.385871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.387838 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.390020 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.390024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.390036 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.390353 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.390693 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.390696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.390698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.390870 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.391290 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.391310 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.391520 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.391669 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.391915 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260924 20:13:14.394156 32104 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40875
I20260924 20:13:14.394310 32104 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:14.394834 32104 heartbeater.cc:511] Master 127.29.83.190:40875 requested a full tablet report, sending...
I20260924 20:13:14.395859 31760 ts_manager.cc:194] Registered new tserver with Master: 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951)
I20260924 20:13:14.396111 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 31973
I20260924 20:13:14.396212 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:14.396499 30030 external_mini_cluster.cc:1428] Setting key 2e60836cfad858da53c43c36f5a80edc
I20260924 20:13:14.396667 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:38701
I20260924 20:13:14.403846 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.396115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:33201 (local address 127.29.83.190:40875)
0924 20:13:14.396226 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.396230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.396508 (+ 278us) server_negotiation.cc:407] Connection header received
0924 20:13:14.396663 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.396666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.396709 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.396799 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.397428 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.398185 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.399215 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.399483 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.399565 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.400705 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.400708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.400710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.400730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.402212 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.402691 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.402693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.402694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.402761 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.403071 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.403073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.403074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.403252 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.403330 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.403635 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.403723 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:13:14.404034 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.396074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.130:33201)
0924 20:13:14.396431 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.396443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.396584 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.396927 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.396930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.397149 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.397297 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.397305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.398340 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.398343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.399071 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.399080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.399590 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.399596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.399710 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.400199 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.400215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.400590 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.402318 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.402322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.402324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.402589 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.402861 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.402866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.402871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.402978 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.403346 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.403352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.403466 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.403593 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.403754 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:14.423012 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40875
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:14.553026 32113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:14.553359 32113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:14.553448 32113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:14.557564 32113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:14.557643 32113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:14.557722 32113 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:13:14.557777 32113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:14.557847 32113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:14.563344 32113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:14.564451 32113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:14.565466 32113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:14.575161 32119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:14.575193 32121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:14.575371 32113 server_base.cc:1034] running on GCE node
W20260924 20:13:14.575177 32118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:14.575837 32113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:14.576458 32113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:14.577641 32113 hybrid_clock.cc:648] HybridClock initialized: now 1790280794577592 us; error 49 us; skew 500 ppm
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:14.580421 32113 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:14.581316 32113 webserver.cc:460] Webserver started at http://127.29.83.131:36927/ using document root <none> and password file <none>
I20260924 20:13:14.581926 32113 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:14.582013 32113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:14.582242 32113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:14.584369 32113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "247efd97d0a94f3fa3566d31b000b8d7"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "c9a6bd20eb9a30c405829446e9615e23"
server_key_iv: "1eebb95ba3156b8ba307f7880641ede6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.584928 32113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "247efd97d0a94f3fa3566d31b000b8d7"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "c9a6bd20eb9a30c405829446e9615e23"
server_key_iv: "1eebb95ba3156b8ba307f7880641ede6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.588865 32113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:13:14.591246 32128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.592427 32113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:14.592564 32113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "247efd97d0a94f3fa3566d31b000b8d7"
format_stamp: "Formatted at 2026-09-24 20:13:14 on dist-test-slave-52fr"
server_key: "c9a6bd20eb9a30c405829446e9615e23"
server_key_iv: "1eebb95ba3156b8ba307f7880641ede6"
server_key_version: "encryptionkey@0"
I20260924 20:13:14.592674 32113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:14.609210 32113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:14.612113 32113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:14.612325 32113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:14.613829 32113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:14.613893 32113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:14.613960 32113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:14.614005 32113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:14.624759 32113 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:37867
I20260924 20:13:14.624832 32243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:37867 every 8 connection(s)
I20260924 20:13:14.625906 32113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:14.628121 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.614539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:53579 (local address 127.29.83.190:40875)
0924 20:13:14.614697 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.614702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.616057 (+ 1355us) server_negotiation.cc:407] Connection header received
0924 20:13:14.617479 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.617483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.617543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.617618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.619416 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.620109 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.621457 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.621748 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.621883 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.624596 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.624600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.624603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.624638 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.626248 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.626743 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.626747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.626748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.626831 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.627166 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.627168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.627169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.627344 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.627420 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.627889 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.628000 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:13:14.628845 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.615366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.131:53579)
0924 20:13:14.615871 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.615936 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.617173 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.617867 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.617880 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.618428 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.619214 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.619238 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.620249 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.620252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.621328 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.621337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.621925 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.621929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.622130 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.622813 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.622839 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.624407 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.626358 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.626364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.626379 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.626630 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.626915 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.626918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.626920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.627086 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.627424 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.627430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.627645 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.627798 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.628040 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260924 20:13:14.630089 32244 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40875
I20260924 20:13:14.630133 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 32113
I20260924 20:13:14.630250 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:14.630262 32244 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:14.630564 30030 external_mini_cluster.cc:1428] Setting key e38c970ac1b01aee2fa8be6cc34b7409
I20260924 20:13:14.630780 32244 heartbeater.cc:511] Master 127.29.83.190:40875 requested a full tablet report, sending...
I20260924 20:13:14.631850 31761 ts_manager.cc:194] Registered new tserver with Master: 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:14.632597 31761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:53579
I20260924 20:13:14.632689 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:13:14.640607 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.632707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:44833 (local address 127.29.83.190:40875)
0924 20:13:14.632921 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.632926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.633138 (+ 212us) server_negotiation.cc:407] Connection header received
0924 20:13:14.633271 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.633275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.633316 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.633401 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.633857 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.634357 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.635499 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.635734 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.635830 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.637073 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.637080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.637086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.637118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.638761 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.639270 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.639273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.639275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.639347 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.639658 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.639661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.639662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.639942 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.640056 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.640327 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.640445 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 20:13:14.640614 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.632637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.29.83.131:44833)
0924 20:13:14.633064 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.633079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.633175 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.633416 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.633419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.633611 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.633700 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.633705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.634476 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.634479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.635362 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.635374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.635856 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.635862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.635975 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.636566 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.636576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.636924 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.638868 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.638874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.638880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.639154 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.639432 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.639438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.639443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.639546 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.640044 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.640047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.640148 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.640263 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.640420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":89,"threads_started":1}
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:14.660075 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.652075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.29.83.190:40875)
0924 20:13:14.652201 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.652204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.652218 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:14.652263 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.652265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.652303 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.652374 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.653051 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.653564 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.654479 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.654694 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.654778 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.657210 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.657213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.657215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.657240 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.658675 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.659049 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.659051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.659053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.659122 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.659355 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.659358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.659359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.659528 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.659628 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.659793 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.659889 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:14.662979 31761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:14.666026 31761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:14.680395 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.674859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.29.83.129:40295)
0924 20:13:14.675482 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.675488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.675507 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:13:14.675569 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.675573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.675817 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.676332 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.676483 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.677622 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.679077 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.679715 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.679847 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.680016 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.680144 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.680244 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":90,"threads_started":1}
I20260924 20:13:14.680395 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.674713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40295 (local address 127.0.0.1:50880)
0924 20:13:14.675104 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.675159 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.675300 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.675988 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.675993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.676017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.676165 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.676171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.677739 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.677745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.678941 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.678950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.679847 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.679851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.679955 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.679997 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.680110 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.680231 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":51,"thread_start_us":104,"threads_started":1}
I20260924 20:13:14.680595 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.675116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:37867 (local address 127.0.0.1:56676)
0924 20:13:14.675735 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.675748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.675835 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.676534 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.676537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.676558 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.676677 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.676685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.677826 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.677830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.679125 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.679135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.680199 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.680205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.680317 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.680341 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.680389 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.680457 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":81,"thread_start_us":89,"threads_started":1}
I20260924 20:13:14.681281 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.675530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.29.83.131:37867)
0924 20:13:14.676068 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.676080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.676103 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:13:14.676171 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.676176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.676382 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.676533 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.676815 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.677701 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.679271 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.680070 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.680763 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.680974 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.681026 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.681116 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":95,"threads_started":1}
I20260924 20:13:14.681479 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.675496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:40951 (local address 127.0.0.1:37006)
0924 20:13:14.676661 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.676674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.676781 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.677204 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.677207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.677226 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.677306 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.677311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.678684 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.678687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.679832 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.679843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.681091 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.681095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.681176 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.681197 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.681273 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.681351 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260924 20:13:14.681905 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.675931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.29.83.130:40951)
0924 20:13:14.676361 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.676374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.676792 (+ 418us) server_negotiation.cc:407] Connection header received
0924 20:13:14.676873 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.676878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.677056 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.677204 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.677416 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.678554 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.679980 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.680808 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.680902 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.681033 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.681496 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.681715 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":110,"threads_started":1}
I20260924 20:13:14.684013 32178 tablet_service.cc:1467] Processing CreateTablet for tablet 13611fc288d9498b9212d279b3795ebc (DEFAULT_TABLE table=test-table [id=748ba6fecfe0417d88f5e983bf4597d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:14.684033 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 13611fc288d9498b9212d279b3795ebc (DEFAULT_TABLE table=test-table [id=748ba6fecfe0417d88f5e983bf4597d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:14.684303 32038 tablet_service.cc:1467] Processing CreateTablet for tablet 13611fc288d9498b9212d279b3795ebc (DEFAULT_TABLE table=test-table [id=748ba6fecfe0417d88f5e983bf4597d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:14.685102 32178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13611fc288d9498b9212d279b3795ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.685114 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13611fc288d9498b9212d279b3795ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.685114 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13611fc288d9498b9212d279b3795ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.691902 32267 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Bootstrap starting.
I20260924 20:13:14.692996 32269 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Bootstrap starting.
I20260924 20:13:14.694075 32268 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Bootstrap starting.
I20260924 20:13:14.694284 32267 tablet_bootstrap.cc:654] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.695042 32269 tablet_bootstrap.cc:654] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.695056 32267 log.cc:824] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:14.695861 32269 log.cc:824] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:14.696144 32268 tablet_bootstrap.cc:654] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.696997 32268 log.cc:824] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:14.697441 32267 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: No bootstrap required, opened a new log
I20260924 20:13:14.697640 32267 ts_tablet_manager.cc:1397] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:13:14.698930 32268 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: No bootstrap required, opened a new log
I20260924 20:13:14.699112 32268 ts_tablet_manager.cc:1397] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:13:14.701642 32267 raft_consensus.cc:348] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.701949 32267 raft_consensus.cc:374] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.702015 32267 raft_consensus.cc:729] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247efd97d0a94f3fa3566d31b000b8d7, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.702472 32267 consensus_queue.cc:260] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.703032 32269 tablet_bootstrap.cc:492] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: No bootstrap required, opened a new log
I20260924 20:13:14.703161 32244 heartbeater.cc:503] Master 127.29.83.190:40875 was elected leader, sending a full tablet report...
I20260924 20:13:14.703207 32269 ts_tablet_manager.cc:1397] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260924 20:13:14.703531 32267 ts_tablet_manager.cc:1428] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:13:14.703467 32268 raft_consensus.cc:348] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.703711 32268 raft_consensus.cc:374] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.703763 32268 raft_consensus.cc:729] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42122042a264557bd70a015a84c173a, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.704186 32268 consensus_queue.cc:260] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.705135 32268 ts_tablet_manager.cc:1428] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 20:13:14.705495 31964 heartbeater.cc:503] Master 127.29.83.190:40875 was elected leader, sending a full tablet report...
I20260924 20:13:14.706671 32269 raft_consensus.cc:348] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.706918 32269 raft_consensus.cc:374] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.706979 32269 raft_consensus.cc:729] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.707405 32269 consensus_queue.cc:260] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.708060 32104 heartbeater.cc:503] Master 127.29.83.190:40875 was elected leader, sending a full tablet report...
I20260924 20:13:14.709218 32269 ts_tablet_manager.cc:1428] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 20:13:14.782918 32275 raft_consensus.cc:482] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:14.783147 32275 raft_consensus.cc:504] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.784498 32275 leader_election.cc:288] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:14.788357 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.784828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:46437 (local address 127.29.83.129:40295)
0924 20:13:14.784953 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.784959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.784977 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:13:14.785042 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.785044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.785093 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.785194 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.785461 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.786057 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.787234 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.787755 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.787876 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.787932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.788177 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.788259 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:14.788355 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.784749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40295 (local address 127.29.83.130:46437)
0924 20:13:14.784856 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.784870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.784960 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.785209 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.785212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.785231 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.785347 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.785359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.786159 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.786164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.787104 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.787113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.787868 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.787872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.788026 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.788050 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.788150 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.788217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":4}
I20260924 20:13:14.789144 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.784756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:37867 (local address 127.29.83.130:36851)
0924 20:13:14.785405 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.785416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.785524 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.785789 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.785792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.785808 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.786278 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.786284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.787069 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.787074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.787910 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.787919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.788759 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.788763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.788836 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.788853 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.788941 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.789006 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":278,"thread_start_us":170,"threads_started":1}
I20260924 20:13:14.789182 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.784935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:36851 (local address 127.29.83.131:37867)
0924 20:13:14.785072 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.785076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.785490 (+ 414us) server_negotiation.cc:407] Connection header received
0924 20:13:14.785624 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.785627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.785669 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.785796 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.786387 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.786965 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.788025 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.788634 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.788723 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.788757 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.788968 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.789075 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:13:14.789260 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13611fc288d9498b9212d279b3795ebc" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a" is_pre_election: true
I20260924 20:13:14.789599 31918 raft_consensus.cc:2454] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:14.790103 31990 leader_election.cc:302] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:14.790333 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13611fc288d9498b9212d279b3795ebc" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7" is_pre_election: true
I20260924 20:13:14.790402 32275 raft_consensus.cc:2781] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:14.790508 32275 raft_consensus.cc:482] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:14.790561 32275 raft_consensus.cc:3037] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.790635 32198 raft_consensus.cc:2454] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:14.791625 32275 raft_consensus.cc:504] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.791963 32275 leader_election.cc:288] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Requested vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:14.792344 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13611fc288d9498b9212d279b3795ebc" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a"
I20260924 20:13:14.792515 31918 raft_consensus.cc:3037] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.793608 31918 raft_consensus.cc:2454] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:14.793968 31990 leader_election.cc:302] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:14.794013 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13611fc288d9498b9212d279b3795ebc" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7"
I20260924 20:13:14.794124 32275 raft_consensus.cc:2781] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:14.794179 32198 raft_consensus.cc:3037] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.794373 32275 raft_consensus.cc:686] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Running, Role: LEADER
I20260924 20:13:14.794950 32275 consensus_queue.cc:237] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.795780 32198 raft_consensus.cc:2454] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:14.798444 31761 catalog_manager.cc:5495] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:13:14 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280793, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:14.852624 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.843606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40875 (local address 127.0.0.1:48320)
0924 20:13:14.843929 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.843941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.844075 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.844325 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.844328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.844776 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:14.844871 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.844877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.845814 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.845819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.846396 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.846404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.846837 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.846840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.846915 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.847650 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:14.847663 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:14.849160 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:14.850945 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.850957 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.850965 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.851199 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.851525 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:14.851528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:14.851530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:14.851609 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:14.851992 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:14.852001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:14.852114 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.852230 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.852426 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260924 20:13:14.852622 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.843615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.29.83.190:40875)
0924 20:13:14.843785 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.843793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.844018 (+ 225us) server_negotiation.cc:407] Connection header received
0924 20:13:14.844180 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.844183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.844224 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.844305 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:14.845129 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.845684 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.846510 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.846730 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.846834 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.849334 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:14.849338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:14.849339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:14.849361 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:14.850822 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.851322 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.851326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.851328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.851430 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:14.851699 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:14.851702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:14.851703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:14.851881 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:14.851978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.852273 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.852486 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":62}
I20260924 20:13:14.857225 31761 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:48320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:13:14.858215 31761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:14.863214 32178 tablet_service.cc:1467] Processing CreateTablet for tablet 175d48109ffb44ebb7b21e0b0b986faf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39b85ef2e98b435d8906dbcffc1e17b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:14.863193 32038 tablet_service.cc:1467] Processing CreateTablet for tablet 175d48109ffb44ebb7b21e0b0b986faf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39b85ef2e98b435d8906dbcffc1e17b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:14.863188 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 175d48109ffb44ebb7b21e0b0b986faf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39b85ef2e98b435d8906dbcffc1e17b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:14.863971 32178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175d48109ffb44ebb7b21e0b0b986faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.863971 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175d48109ffb44ebb7b21e0b0b986faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.863986 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175d48109ffb44ebb7b21e0b0b986faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:14.866726 32268 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: Bootstrap starting.
I20260924 20:13:14.866806 32269 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Bootstrap starting.
I20260924 20:13:14.867612 32268 tablet_bootstrap.cc:654] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.867719 32269 tablet_bootstrap.cc:654] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.868705 32267 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: Bootstrap starting.
I20260924 20:13:14.868896 32268 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: No bootstrap required, opened a new log
I20260924 20:13:14.868968 32268 ts_tablet_manager.cc:1397] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 20:13:14.869280 32269 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: No bootstrap required, opened a new log
I20260924 20:13:14.869360 32269 ts_tablet_manager.cc:1397] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 20:13:14.869450 32268 raft_consensus.cc:348] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.869562 32268 raft_consensus.cc:374] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.869599 32268 raft_consensus.cc:729] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42122042a264557bd70a015a84c173a, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.869694 32267 tablet_bootstrap.cc:654] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:14.869740 32268 consensus_queue.cc:260] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.869735 32269 raft_consensus.cc:348] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.869892 32269 raft_consensus.cc:374] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.869925 32269 raft_consensus.cc:729] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.870038 32269 consensus_queue.cc:260] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.870074 32268 ts_tablet_manager.cc:1428] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:13:14.870051 32274 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.870193 32274 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.870288 32269 ts_tablet_manager.cc:1428] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:14.870718 32275 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.870836 32275 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.872609 32267 tablet_bootstrap.cc:492] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: No bootstrap required, opened a new log
I20260924 20:13:14.872697 32267 ts_tablet_manager.cc:1397] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 20:13:14.873093 32267 raft_consensus.cc:348] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.873201 32267 raft_consensus.cc:374] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:14.873229 32267 raft_consensus.cc:729] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247efd97d0a94f3fa3566d31b000b8d7, State: Initialized, Role: FOLLOWER
I20260924 20:13:14.873334 32267 consensus_queue.cc:260] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.873553 32267 ts_tablet_manager.cc:1428] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:13:14.873572 32273 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.873713 32273 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 20:13:14.877228 32245 tablet.cc:2369] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:14.891358 32105 tablet.cc:2369] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:14.891839 32274 raft_consensus.cc:482] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:14.891976 32274 raft_consensus.cc:504] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.893049 32274 leader_election.cc:288] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:14.897683 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.893420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:32843 (local address 127.29.83.131:37867)
0924 20:13:14.893547 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.893551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.893832 (+ 281us) server_negotiation.cc:407] Connection header received
0924 20:13:14.893949 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.893952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.894006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.894076 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.894471 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.895153 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.896516 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.897015 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.897147 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.897188 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.897462 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.897550 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:13:14.897723 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.893342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:37867 (local address 127.29.83.129:32843)
0924 20:13:14.893698 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.893735 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.893850 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.894086 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.894089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.894109 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.894338 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.894349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.895262 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.895266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.896381 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.896391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.897212 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.897216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.897330 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.897357 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.897431 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.897493 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":89,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260924 20:13:14.897781 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.893420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:36315 (local address 127.29.83.130:40951)
0924 20:13:14.893807 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.893814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.893831 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:13:14.893898 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.893901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.893954 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.894045 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:14.894792 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.895536 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.896675 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.897213 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.897330 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.897380 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.897551 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.897638 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260924 20:13:14.897778 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.893342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:40951 (local address 127.29.83.129:36315)
0924 20:13:14.893647 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.893662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.893804 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.894072 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.894077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.894095 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:14.894683 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.894690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.895632 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.895636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.896509 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.896521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.897323 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.897326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.897422 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.897450 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.897526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.897591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":34,"thread_start_us":127,"threads_started":1}
I20260924 20:13:14.898288 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175d48109ffb44ebb7b21e0b0b986faf" candidate_uuid: "d42122042a264557bd70a015a84c173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7" is_pre_election: true
I20260924 20:13:14.898442 32198 raft_consensus.cc:2454] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42122042a264557bd70a015a84c173a in term 0.
I20260924 20:13:14.898844 31849 leader_election.cc:302] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:14.898844 32058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175d48109ffb44ebb7b21e0b0b986faf" candidate_uuid: "d42122042a264557bd70a015a84c173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" is_pre_election: true
I20260924 20:13:14.899082 32058 raft_consensus.cc:2454] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42122042a264557bd70a015a84c173a in term 0.
I20260924 20:13:14.899358 32274 raft_consensus.cc:2781] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:14.899410 32274 raft_consensus.cc:482] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:14.899436 32274 raft_consensus.cc:3037] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.900352 32274 raft_consensus.cc:504] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:14.900679 32274 leader_election.cc:288] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [CANDIDATE]: Term 1 election: Requested vote from peers 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:14.900995 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175d48109ffb44ebb7b21e0b0b986faf" candidate_uuid: "d42122042a264557bd70a015a84c173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7"
I20260924 20:13:14.901062 32058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175d48109ffb44ebb7b21e0b0b986faf" candidate_uuid: "d42122042a264557bd70a015a84c173a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2"
I20260924 20:13:14.901130 32198 raft_consensus.cc:3037] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.901211 32058 raft_consensus.cc:3037] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:14.902144 32198 raft_consensus.cc:2454] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42122042a264557bd70a015a84c173a in term 1.
I20260924 20:13:14.902293 32058 raft_consensus.cc:2454] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42122042a264557bd70a015a84c173a in term 1.
I20260924 20:13:14.902411 31849 leader_election.cc:302] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:14.902591 32274 raft_consensus.cc:2781] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:14.902832 32274 raft_consensus.cc:686] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [term 1 LEADER]: Becoming Leader. State: Replica: d42122042a264557bd70a015a84c173a, State: Running, Role: LEADER
I20260924 20:13:14.903218 32274 consensus_queue.cc:237] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
W20260924 20:13:14.903967 31965 tablet.cc:2369] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:14.905596 32290 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: TxnStatusTablet state changed. Reason: New leader d42122042a264557bd70a015a84c173a. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.905746 32290 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:14.906314 32292 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:13:14.906510 32292 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:13:14.906955 31759 catalog_manager.cc:5495] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a reported cstate change: term changed from 0 to 1, leader changed from <none> to d42122042a264557bd70a015a84c173a (127.29.83.129). New cstate: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:14.924985 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.920943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.29.83.129:40295)
0924 20:13:14.921056 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.921060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.921143 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:13:14.921356 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.921358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.921411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.921495 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:14.921809 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.922417 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.923617 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.923828 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.923908 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.924232 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.924633 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.924732 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.924824 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.924876 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:13:14.925051 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.920841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40295 (local address 127.0.0.1:50882)
0924 20:13:14.921072 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.921087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.921208 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.921536 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.921539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.921547 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:14.921658 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.921663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.922515 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.922519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.923497 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.923508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.923933 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.923936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.924034 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.924087 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.924725 (+ 638us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.924741 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.924816 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.924905 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 20:13:14.930478 32198 raft_consensus.cc:1261] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Refusing update from remote peer d42122042a264557bd70a015a84c173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:14.930558 32058 raft_consensus.cc:1261] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Refusing update from remote peer d42122042a264557bd70a015a84c173a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:14.931200 32290 consensus_queue.cc:1035] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:14.931483 32274 consensus_queue.cc:1035] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:14.934621 32275 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: TxnStatusTablet state changed. Reason: New leader d42122042a264557bd70a015a84c173a. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.934768 32275 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.935554 32273 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: TxnStatusTablet state changed. Reason: New leader d42122042a264557bd70a015a84c173a. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.935694 32273 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.938374 32290 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.938510 32290 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:14.939800 32275 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.939917 32275 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.940708 32296 mvcc.cc:205] Tried to move back new op lower bound from 7332990136028110848 to 7332990135926829056. Current Snapshot: MvccSnapshot[applied={T|T < 7332990136028110848}]
I20260924 20:13:14.948006 32290 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.948143 32290 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P d42122042a264557bd70a015a84c173a: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:14.950703 32273 tablet_replica.cc:439] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42122042a264557bd70a015a84c173a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } } }
I20260924 20:13:14.950827 32273 tablet_replica.cc:442] T 175d48109ffb44ebb7b21e0b0b986faf P 247efd97d0a94f3fa3566d31b000b8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:14.951179 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.944346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.29.83.130:40951)
0924 20:13:14.944737 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.944741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.944753 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:14.944819 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.944822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.944879 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.944982 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:14.945249 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.946048 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.947206 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.947548 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.947646 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.949885 (+ 2239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.950395 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.950518 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.950982 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.951042 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260924 20:13:14.961951 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.958443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40295 (local address 127.29.83.130:35501)
0924 20:13:14.958710 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.958723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.958817 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.959032 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.959034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.959039 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:14.959136 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.959140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.960088 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.960090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.960689 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.960696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.961264 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.961268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.961334 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.961372 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.961669 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.961673 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.961719 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.961830 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260924 20:13:14.961943 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.958527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:35501 (local address 127.29.83.129:40295)
0924 20:13:14.958647 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.958652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.958797 (+ 145us) server_negotiation.cc:407] Connection header received
0924 20:13:14.958892 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.958895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.958946 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.959026 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:14.959230 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.959998 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.960798 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.961144 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.961242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.961512 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.961592 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.961661 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.961733 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.961810 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:14.969015 31918 raft_consensus.cc:1261] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:14.969691 32275 consensus_queue.cc:1035] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:14.969817 32198 raft_consensus.cc:1261] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:14.970181 32275 consensus_queue.cc:1035] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:14.974763 32295 mvcc.cc:205] Tried to move back new op lower bound from 7332990136187547648 to 7332990135483449344. Current Snapshot: MvccSnapshot[applied={T|T < 7332990136187547648}]
I20260924 20:13:14.976847 32297 mvcc.cc:205] Tried to move back new op lower bound from 7332990136187547648 to 7332990135483449344. Current Snapshot: MvccSnapshot[applied={T|T < 7332990136187547648}]
I20260924 20:13:14.992908 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.989239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:40951 (local address 127.29.83.129:44521)
0924 20:13:14.989568 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:14.989583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:14.989690 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:14.989956 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:14.989959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:14.989967 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:14.990230 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.990237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.990963 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.990967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.991742 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:14.991751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.992180 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.992184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:14.992286 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.992326 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.992613 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.992620 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:13:14.992680 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:13:14.992752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
I20260924 20:13:14.992884 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:14.989338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:44521 (local address 127.29.83.130:40951)
0924 20:13:14.989457 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:14.989464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:14.989652 (+ 188us) server_negotiation.cc:407] Connection header received
0924 20:13:14.989828 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:14.989830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:14.989869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:14.989930 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:14.990342 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.990848 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.991854 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:14.992063 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:14.992170 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:14.992454 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:14.992521 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:14.992598 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:14.992688 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 20:13:14.992745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:13:15.015780 31759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:15.015969 31759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:15.018431 31759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Sending DeleteTablet for 3 replicas of tablet 13611fc288d9498b9212d279b3795ebc
I20260924 20:13:15.019251 31898 tablet_service.cc:1514] Processing DeleteTablet for tablet 13611fc288d9498b9212d279b3795ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:15 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:50880
I20260924 20:13:15.019395 32038 tablet_service.cc:1514] Processing DeleteTablet for tablet 13611fc288d9498b9212d279b3795ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:15 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 20:13:15.020061 32316 tablet_replica.cc:330] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: stopping tablet replica
I20260924 20:13:15.020102 32178 tablet_service.cc:1514] Processing DeleteTablet for tablet 13611fc288d9498b9212d279b3795ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:15 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:56676
I20260924 20:13:15.020501 32316 raft_consensus.cc:2229] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:15.020927 32316 raft_consensus.cc:2258] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:15.021184 32317 tablet_replica.cc:330] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: stopping tablet replica
I20260924 20:13:15.021364 31759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:13:15.021481 32317 raft_consensus.cc:2229] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:15.021663 32317 raft_consensus.cc:2258] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:13:15.022194 31759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:15.023277 32317 ts_tablet_manager.cc:1905] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:15.027344 32317 ts_tablet_manager.cc:1918] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:15.027448 32317 log.cc:1196] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/13611fc288d9498b9212d279b3795ebc
I20260924 20:13:15.027786 32317 ts_tablet_manager.cc:1939] T 13611fc288d9498b9212d279b3795ebc P 247efd97d0a94f3fa3566d31b000b8d7: Deleting consensus metadata
I20260924 20:13:15.028810 32178 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b (DEFAULT_TABLE table=test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:15.029040 32178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2f8fa7efc54316a0a5769ad462147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:15.030004 31745 catalog_manager.cc:4841] TS 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867): tablet 13611fc288d9498b9212d279b3795ebc (table test-table [id=748ba6fecfe0417d88f5e983bf4597d5]) successfully deleted
I20260924 20:13:15.030174 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b (DEFAULT_TABLE table=test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:15.030483 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2f8fa7efc54316a0a5769ad462147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:15.032500 32268 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Bootstrap starting.
I20260924 20:13:15.033397 32268 tablet_bootstrap.cc:654] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:15.034873 32267 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Bootstrap starting.
I20260924 20:13:15.035533 32268 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: No bootstrap required, opened a new log
I20260924 20:13:15.035609 32268 ts_tablet_manager.cc:1397] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 20:13:15.035856 32267 tablet_bootstrap.cc:654] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:15.036123 32268 raft_consensus.cc:348] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.036267 32268 raft_consensus.cc:374] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:15.036307 32268 raft_consensus.cc:729] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42122042a264557bd70a015a84c173a, State: Initialized, Role: FOLLOWER
I20260924 20:13:15.036444 32268 consensus_queue.cc:260] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.036723 32268 ts_tablet_manager.cc:1428] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:13:15.028484 32038 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b (DEFAULT_TABLE table=test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:15.037142 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2f8fa7efc54316a0a5769ad462147b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:15.039700 32269 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Bootstrap starting.
I20260924 20:13:15.040542 32267 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: No bootstrap required, opened a new log
I20260924 20:13:15.040637 32267 ts_tablet_manager.cc:1397] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 20:13:15.040661 32269 tablet_bootstrap.cc:654] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:15.041384 32267 raft_consensus.cc:348] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.041497 32267 raft_consensus.cc:374] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:15.041519 32267 raft_consensus.cc:729] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247efd97d0a94f3fa3566d31b000b8d7, State: Initialized, Role: FOLLOWER
I20260924 20:13:15.041651 32267 consensus_queue.cc:260] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.041963 32267 ts_tablet_manager.cc:1428] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:15.042559 32315 tablet_replica.cc:330] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: stopping tablet replica
I20260924 20:13:15.043025 32315 raft_consensus.cc:2229] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:15.043340 32315 raft_consensus.cc:2258] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:15.044534 32269 tablet_bootstrap.cc:492] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: No bootstrap required, opened a new log
I20260924 20:13:15.044622 32269 ts_tablet_manager.cc:1397] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 20:13:15.044744 32315 ts_tablet_manager.cc:1905] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:15.045090 32269 raft_consensus.cc:348] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.045238 32269 raft_consensus.cc:374] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:15.045271 32269 raft_consensus.cc:729] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Initialized, Role: FOLLOWER
I20260924 20:13:15.045395 32269 consensus_queue.cc:260] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.045745 32269 ts_tablet_manager.cc:1428] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:15.047968 32315 ts_tablet_manager.cc:1918] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:15.048067 32315 log.cc:1196] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/13611fc288d9498b9212d279b3795ebc
I20260924 20:13:15.048420 32315 ts_tablet_manager.cc:1939] T 13611fc288d9498b9212d279b3795ebc P d42122042a264557bd70a015a84c173a: Deleting consensus metadata
I20260924 20:13:15.049789 31746 catalog_manager.cc:4841] TS d42122042a264557bd70a015a84c173a (127.29.83.129:40295): tablet 13611fc288d9498b9212d279b3795ebc (table test-table [id=748ba6fecfe0417d88f5e983bf4597d5]) successfully deleted
I20260924 20:13:15.049906 32316 ts_tablet_manager.cc:1905] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:15.052938 32316 ts_tablet_manager.cc:1918] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:15.053058 32316 log.cc:1196] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/13611fc288d9498b9212d279b3795ebc
I20260924 20:13:15.053416 32316 ts_tablet_manager.cc:1939] T 13611fc288d9498b9212d279b3795ebc P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting consensus metadata
I20260924 20:13:15.054692 31746 catalog_manager.cc:4841] TS 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951): tablet 13611fc288d9498b9212d279b3795ebc (table test-table [id=748ba6fecfe0417d88f5e983bf4597d5]) successfully deleted
I20260924 20:13:15.443928 32310 raft_consensus.cc:482] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:15.444098 32310 raft_consensus.cc:504] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.444615 32310 leader_election.cc:288] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:15.445114 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2f8fa7efc54316a0a5769ad462147b" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a" is_pre_election: true
I20260924 20:13:15.445112 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2f8fa7efc54316a0a5769ad462147b" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7" is_pre_election: true
I20260924 20:13:15.445283 32198 raft_consensus.cc:2454] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:15.445314 31918 raft_consensus.cc:2454] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:15.445618 31989 leader_election.cc:302] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, 6cdd7b4a69ff47bb8ebdfcabd2058cf2; no voters:
I20260924 20:13:15.445837 32310 raft_consensus.cc:2781] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:15.445894 32310 raft_consensus.cc:482] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:15.445921 32310 raft_consensus.cc:3037] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:15.446957 32310 raft_consensus.cc:504] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.447326 32310 leader_election.cc:288] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Requested vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:15.447671 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2f8fa7efc54316a0a5769ad462147b" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a"
I20260924 20:13:15.447707 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2f8fa7efc54316a0a5769ad462147b" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7"
I20260924 20:13:15.447822 31918 raft_consensus.cc:3037] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:15.447830 32198 raft_consensus.cc:3037] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:15.448727 32198 raft_consensus.cc:2454] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:15.448917 31918 raft_consensus.cc:2454] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:15.449026 31989 leader_election.cc:302] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, 6cdd7b4a69ff47bb8ebdfcabd2058cf2; no voters:
I20260924 20:13:15.449218 32310 raft_consensus.cc:2781] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:15.449325 32310 raft_consensus.cc:686] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Running, Role: LEADER
I20260924 20:13:15.449470 32310 consensus_queue.cc:237] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:15.451280 31759 catalog_manager.cc:5495] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:15.496600 31918 raft_consensus.cc:1261] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:15.496621 32198 raft_consensus.cc:1261] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:15.496976 32310 consensus_queue.cc:1035] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:15.497128 32275 consensus_queue.cc:1035] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:13:15.510532 32296 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:13:17.062577 31760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:17.062729 31760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:17.065102 31760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Sending DeleteTablet for 3 replicas of tablet 8c2f8fa7efc54316a0a5769ad462147b
I20260924 20:13:17.065696 31898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:50880
I20260924 20:13:17.065783 32038 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 20:13:17.065851 32178 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c2f8fa7efc54316a0a5769ad462147b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:56676
I20260924 20:13:17.066169 32349 tablet_replica.cc:330] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: stopping tablet replica
I20260924 20:13:17.066198 32350 tablet_replica.cc:330] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: stopping tablet replica
I20260924 20:13:17.066340 32350 raft_consensus.cc:2229] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:17.066370 32349 raft_consensus.cc:2229] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:17.066476 32350 raft_consensus.cc:2258] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:17.066664 32349 raft_consensus.cc:2258] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:17.066740 32348 tablet_replica.cc:330] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: stopping tablet replica
I20260924 20:13:17.066895 32348 raft_consensus.cc:2229] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:17.067023 32348 raft_consensus.cc:2258] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:17.067576 32350 ts_tablet_manager.cc:1905] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.067704 32348 ts_tablet_manager.cc:1905] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.069741 32349 ts_tablet_manager.cc:1905] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.070353 32350 ts_tablet_manager.cc:1918] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:13:17.070446 32350 log.cc:1196] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/8c2f8fa7efc54316a0a5769ad462147b
I20260924 20:13:17.070621 32348 ts_tablet_manager.cc:1918] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:13:17.070739 32350 ts_tablet_manager.cc:1939] T 8c2f8fa7efc54316a0a5769ad462147b P 247efd97d0a94f3fa3566d31b000b8d7: Deleting consensus metadata
I20260924 20:13:17.070739 32348 log.cc:1196] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/8c2f8fa7efc54316a0a5769ad462147b
I20260924 20:13:17.071144 32348 ts_tablet_manager.cc:1939] T 8c2f8fa7efc54316a0a5769ad462147b P d42122042a264557bd70a015a84c173a: Deleting consensus metadata
I20260924 20:13:17.071556 31745 catalog_manager.cc:4841] TS 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867): tablet 8c2f8fa7efc54316a0a5769ad462147b (table test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]) successfully deleted
I20260924 20:13:17.071885 31746 catalog_manager.cc:4841] TS d42122042a264557bd70a015a84c173a (127.29.83.129:40295): tablet 8c2f8fa7efc54316a0a5769ad462147b (table test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]) successfully deleted
I20260924 20:13:17.072899 32349 ts_tablet_manager.cc:1918] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:13:17.072981 32349 log.cc:1196] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/8c2f8fa7efc54316a0a5769ad462147b
I20260924 20:13:17.073247 32349 ts_tablet_manager.cc:1939] T 8c2f8fa7efc54316a0a5769ad462147b P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting consensus metadata
I20260924 20:13:17.073864 31746 catalog_manager.cc:4841] TS 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951): tablet 8c2f8fa7efc54316a0a5769ad462147b (table test-table [id=b3ba778db4dc4c9fa065c6b0cbafc627]) successfully deleted
Sep 24 20:13:17 dist-test-slave-52fr krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:13:17 dist-test-slave-52fr krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:17.096014 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:17.086191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.29.83.190:40875)
0924 20:13:17.086512 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:17.086519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:17.086535 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:13:17.086734 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:17.086738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:17.086800 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:17.086888 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:17.087775 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:17.088544 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:17.089488 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:17.089829 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:17.089945 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:17.092535 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:17.092538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:17.092540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:17.092567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:17.094160 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:17.094698 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:17.094701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:17.094702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:17.094781 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:17.095095 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:17.095098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:17.095099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:17.095288 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:17.095395 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:17.095681 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 20:13:17.095790 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260924 20:13:17.098873 31760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:17.099296 31760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:17.103871 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 27724ac88ea14fc3a4d32e01957201ad (DEFAULT_TABLE table=test-table [id=59bfcba0c19649f3913155bb53da6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.103930 32038 tablet_service.cc:1467] Processing CreateTablet for tablet 27724ac88ea14fc3a4d32e01957201ad (DEFAULT_TABLE table=test-table [id=59bfcba0c19649f3913155bb53da6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.104193 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27724ac88ea14fc3a4d32e01957201ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.104218 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27724ac88ea14fc3a4d32e01957201ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.104463 32178 tablet_service.cc:1467] Processing CreateTablet for tablet 27724ac88ea14fc3a4d32e01957201ad (DEFAULT_TABLE table=test-table [id=59bfcba0c19649f3913155bb53da6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.104748 32178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27724ac88ea14fc3a4d32e01957201ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.106642 32359 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Bootstrap starting.
I20260924 20:13:17.107767 32361 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Bootstrap starting.
I20260924 20:13:17.107846 32360 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Bootstrap starting.
I20260924 20:13:17.108045 32359 tablet_bootstrap.cc:654] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.108644 32361 tablet_bootstrap.cc:654] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.108644 32360 tablet_bootstrap.cc:654] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.109071 32359 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: No bootstrap required, opened a new log
I20260924 20:13:17.109182 32359 ts_tablet_manager.cc:1397] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:17.109678 32359 raft_consensus.cc:348] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.109850 32359 raft_consensus.cc:374] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.109894 32359 raft_consensus.cc:729] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42122042a264557bd70a015a84c173a, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.109956 32361 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: No bootstrap required, opened a new log
I20260924 20:13:17.110046 32361 ts_tablet_manager.cc:1397] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:17.110040 32359 consensus_queue.cc:260] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.110319 32359 ts_tablet_manager.cc:1428] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.110517 32361 raft_consensus.cc:348] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.110671 32361 raft_consensus.cc:374] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.110698 32361 raft_consensus.cc:729] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247efd97d0a94f3fa3566d31b000b8d7, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.110831 32361 consensus_queue.cc:260] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.110877 32360 tablet_bootstrap.cc:492] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: No bootstrap required, opened a new log
I20260924 20:13:17.110965 32360 ts_tablet_manager.cc:1397] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 20:13:17.111222 32361 ts_tablet_manager.cc:1428] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.111354 32360 raft_consensus.cc:348] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.111483 32360 raft_consensus.cc:374] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.111521 32360 raft_consensus.cc:729] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.111665 32360 consensus_queue.cc:260] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.112037 32360 ts_tablet_manager.cc:1428] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.392776 32342 raft_consensus.cc:482] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:17.392922 32342 raft_consensus.cc:504] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.393378 32342 leader_election.cc:288] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:17.393846 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27724ac88ea14fc3a4d32e01957201ad" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a" is_pre_election: true
I20260924 20:13:17.393842 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27724ac88ea14fc3a4d32e01957201ad" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7" is_pre_election: true
I20260924 20:13:17.394002 32198 raft_consensus.cc:2454] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:17.394027 31918 raft_consensus.cc:2454] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:17.394357 31989 leader_election.cc:302] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, 6cdd7b4a69ff47bb8ebdfcabd2058cf2; no voters:
I20260924 20:13:17.394622 32342 raft_consensus.cc:2781] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:17.394691 32342 raft_consensus.cc:482] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:17.394743 32342 raft_consensus.cc:3037] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.395908 32342 raft_consensus.cc:504] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.396270 32342 leader_election.cc:288] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Requested vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:17.396624 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27724ac88ea14fc3a4d32e01957201ad" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a"
I20260924 20:13:17.396631 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27724ac88ea14fc3a4d32e01957201ad" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7"
I20260924 20:13:17.396762 31918 raft_consensus.cc:3037] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.396755 32198 raft_consensus.cc:3037] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.397760 31918 raft_consensus.cc:2454] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:17.397921 32198 raft_consensus.cc:2454] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:17.398128 31990 leader_election.cc:302] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:17.398366 32342 raft_consensus.cc:2781] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:17.398460 32342 raft_consensus.cc:686] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Running, Role: LEADER
I20260924 20:13:17.398597 32342 consensus_queue.cc:237] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.400236 31759 catalog_manager.cc:5495] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:17.482945 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:17.479650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.29.83.130:40951)
0924 20:13:17.479917 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:17.479923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:17.479935 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:17.479976 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:17.479978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:17.480017 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:17.480090 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:17.480330 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:17.480846 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:17.481711 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:17.482039 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:17.482161 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:17.482442 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:17.482522 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:17.482593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:17.482691 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 20:13:17.482740 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260924 20:13:17.489548 31918 raft_consensus.cc:1261] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:17.489902 32198 raft_consensus.cc:1261] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:17.490028 32339 consensus_queue.cc:1035] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:17.490326 32342 consensus_queue.cc:1035] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:17.521474 31760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:17.521695 31760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:17.523819 31760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Sending DeleteTablet for 3 replicas of tablet 27724ac88ea14fc3a4d32e01957201ad
I20260924 20:13:17.524430 31898 tablet_service.cc:1514] Processing DeleteTablet for tablet 27724ac88ea14fc3a4d32e01957201ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:50880
I20260924 20:13:17.524490 32178 tablet_service.cc:1514] Processing DeleteTablet for tablet 27724ac88ea14fc3a4d32e01957201ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:56676
I20260924 20:13:17.524659 32348 tablet_replica.cc:330] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: stopping tablet replica
I20260924 20:13:17.524667 32350 tablet_replica.cc:330] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: stopping tablet replica
I20260924 20:13:17.524801 32348 raft_consensus.cc:2229] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:17.524818 32350 raft_consensus.cc:2229] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:17.524940 32350 raft_consensus.cc:2258] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:17.524935 32348 raft_consensus.cc:2258] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:17.525120 32038 tablet_service.cc:1514] Processing DeleteTablet for tablet 27724ac88ea14fc3a4d32e01957201ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:17 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 20:13:17.525295 32349 tablet_replica.cc:330] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: stopping tablet replica
I20260924 20:13:17.525424 32349 raft_consensus.cc:2229] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:17.525439 31760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:13:17.525628 32349 raft_consensus.cc:2258] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:13:17.525892 31760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:17.526067 32350 ts_tablet_manager.cc:1905] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.526292 32349 ts_tablet_manager.cc:1905] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.526847 32348 ts_tablet_manager.cc:1905] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:17.528641 32349 ts_tablet_manager.cc:1918] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:17.528657 32350 ts_tablet_manager.cc:1918] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:17.528714 32349 log.cc:1196] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/27724ac88ea14fc3a4d32e01957201ad
I20260924 20:13:17.528728 32350 log.cc:1196] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/27724ac88ea14fc3a4d32e01957201ad
I20260924 20:13:17.528978 32350 ts_tablet_manager.cc:1939] T 27724ac88ea14fc3a4d32e01957201ad P 247efd97d0a94f3fa3566d31b000b8d7: Deleting consensus metadata
I20260924 20:13:17.528978 32349 ts_tablet_manager.cc:1939] T 27724ac88ea14fc3a4d32e01957201ad P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting consensus metadata
I20260924 20:13:17.529214 32348 ts_tablet_manager.cc:1918] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:17.529287 32348 log.cc:1196] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/27724ac88ea14fc3a4d32e01957201ad
I20260924 20:13:17.529542 32348 ts_tablet_manager.cc:1939] T 27724ac88ea14fc3a4d32e01957201ad P d42122042a264557bd70a015a84c173a: Deleting consensus metadata
I20260924 20:13:17.529721 31746 catalog_manager.cc:4841] TS 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951): tablet 27724ac88ea14fc3a4d32e01957201ad (table test-table [id=59bfcba0c19649f3913155bb53da6028]) successfully deleted
I20260924 20:13:17.530206 31746 catalog_manager.cc:4841] TS d42122042a264557bd70a015a84c173a (127.29.83.129:40295): tablet 27724ac88ea14fc3a4d32e01957201ad (table test-table [id=59bfcba0c19649f3913155bb53da6028]) successfully deleted
I20260924 20:13:17.530210 31745 catalog_manager.cc:4841] TS 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867): tablet 27724ac88ea14fc3a4d32e01957201ad (table test-table [id=59bfcba0c19649f3913155bb53da6028]) successfully deleted
I20260924 20:13:17.532060 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 (DEFAULT_TABLE table=test-table [id=af310b61d40e4859b0a3bd204155d570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.532109 32038 tablet_service.cc:1467] Processing CreateTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 (DEFAULT_TABLE table=test-table [id=af310b61d40e4859b0a3bd204155d570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.532338 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.532348 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.532354 32178 tablet_service.cc:1467] Processing CreateTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 (DEFAULT_TABLE table=test-table [id=af310b61d40e4859b0a3bd204155d570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:17.532586 32178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:17.534200 32360 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Bootstrap starting.
I20260924 20:13:17.534739 32359 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a: Bootstrap starting.
I20260924 20:13:17.534962 32360 tablet_bootstrap.cc:654] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.535037 32361 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Bootstrap starting.
I20260924 20:13:17.535601 32359 tablet_bootstrap.cc:654] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.535912 32361 tablet_bootstrap.cc:654] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:17.536121 32360 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: No bootstrap required, opened a new log
I20260924 20:13:17.536214 32360 ts_tablet_manager.cc:1397] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:17.536626 32360 raft_consensus.cc:348] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.536758 32360 raft_consensus.cc:374] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.536798 32360 raft_consensus.cc:729] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.536881 32359 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a: No bootstrap required, opened a new log
I20260924 20:13:17.536968 32359 ts_tablet_manager.cc:1397] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:17.536922 32360 consensus_queue.cc:260] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.537197 32360 ts_tablet_manager.cc:1428] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.537382 32359 raft_consensus.cc:348] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.537492 32359 raft_consensus.cc:374] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.537524 32359 raft_consensus.cc:729] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42122042a264557bd70a015a84c173a, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.537524 32361 tablet_bootstrap.cc:492] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: No bootstrap required, opened a new log
I20260924 20:13:17.537607 32361 ts_tablet_manager.cc:1397] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:17.537643 32359 consensus_queue.cc:260] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.537930 32359 ts_tablet_manager.cc:1428] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.538089 32361 raft_consensus.cc:348] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.538215 32361 raft_consensus.cc:374] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:17.538250 32361 raft_consensus.cc:729] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247efd97d0a94f3fa3566d31b000b8d7, State: Initialized, Role: FOLLOWER
I20260924 20:13:17.538370 32361 consensus_queue.cc:260] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.538655 32361 ts_tablet_manager.cc:1428] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:17.695401 32342 raft_consensus.cc:482] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:17.695524 32342 raft_consensus.cc:504] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.695930 32342 leader_election.cc:288] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:17.696353 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c01015ca3e41f2b4c87a6c3b0f47c3" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a" is_pre_election: true
I20260924 20:13:17.696408 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c01015ca3e41f2b4c87a6c3b0f47c3" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7" is_pre_election: true
I20260924 20:13:17.696512 31918 raft_consensus.cc:2454] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:17.696609 32198 raft_consensus.cc:2454] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 0.
I20260924 20:13:17.696842 31990 leader_election.cc:302] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, d42122042a264557bd70a015a84c173a; no voters:
I20260924 20:13:17.697003 32342 raft_consensus.cc:2781] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:17.697060 32342 raft_consensus.cc:482] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:17.697108 32342 raft_consensus.cc:3037] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.698122 32342 raft_consensus.cc:504] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.698446 32342 leader_election.cc:288] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Requested vote from peers d42122042a264557bd70a015a84c173a (127.29.83.129:40295), 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867)
I20260924 20:13:17.698712 31918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c01015ca3e41f2b4c87a6c3b0f47c3" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42122042a264557bd70a015a84c173a"
I20260924 20:13:17.698776 32198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c01015ca3e41f2b4c87a6c3b0f47c3" candidate_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247efd97d0a94f3fa3566d31b000b8d7"
I20260924 20:13:17.698832 31918 raft_consensus.cc:3037] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.698884 32198 raft_consensus.cc:3037] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:17.699942 32198 raft_consensus.cc:2454] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:17.699941 31918 raft_consensus.cc:2454] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cdd7b4a69ff47bb8ebdfcabd2058cf2 in term 1.
I20260924 20:13:17.700194 31989 leader_election.cc:302] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247efd97d0a94f3fa3566d31b000b8d7, 6cdd7b4a69ff47bb8ebdfcabd2058cf2; no voters:
I20260924 20:13:17.700353 32342 raft_consensus.cc:2781] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:17.700434 32342 raft_consensus.cc:686] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdd7b4a69ff47bb8ebdfcabd2058cf2, State: Running, Role: LEADER
I20260924 20:13:17.700590 32342 consensus_queue.cc:237] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } }
I20260924 20:13:17.702215 31759 catalog_manager.cc:5495] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdd7b4a69ff47bb8ebdfcabd2058cf2" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:17.740602 32198 raft_consensus.cc:1261] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:17.740739 31918 raft_consensus.cc:1261] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P d42122042a264557bd70a015a84c173a [term 1 FOLLOWER]: Refusing update from remote peer 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:17.741027 32339 consensus_queue.cc:1035] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247efd97d0a94f3fa3566d31b000b8d7" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:17.741189 32342 consensus_queue.cc:1035] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42122042a264557bd70a015a84c173a" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:13:17.753476 32329 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:13:19.302326 31761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:19.302470 31761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:19.304780 31761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7035d8d5bb74dc98ff0e01d32ea3f17: Sending DeleteTablet for 3 replicas of tablet 67c01015ca3e41f2b4c87a6c3b0f47c3
I20260924 20:13:19.305290 31898 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:19 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:50880
I20260924 20:13:19.305313 32178 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:19 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:56676
I20260924 20:13:19.305605 32038 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:19 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260924 20:13:19.305678 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31833
I20260924 20:13:19.306449 32402 tablet_replica.cc:330] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: stopping tablet replica
I20260924 20:13:19.306605 32402 raft_consensus.cc:2229] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:19.306803 32402 raft_consensus.cc:2258] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:19.307046 32401 tablet_replica.cc:330] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: stopping tablet replica
I20260924 20:13:19.307205 32401 raft_consensus.cc:2229] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:19.307327 32401 raft_consensus.cc:2258] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:19.307616 32402 ts_tablet_manager.cc:1905] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:19.307966 32401 ts_tablet_manager.cc:1905] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:19.310113 32402 ts_tablet_manager.cc:1918] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:13:19.310212 32402 log.cc:1196] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/67c01015ca3e41f2b4c87a6c3b0f47c3
I20260924 20:13:19.310526 32402 ts_tablet_manager.cc:1939] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 6cdd7b4a69ff47bb8ebdfcabd2058cf2: Deleting consensus metadata
I20260924 20:13:19.311062 32401 ts_tablet_manager.cc:1918] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:13:19.311154 32401 log.cc:1196] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/67c01015ca3e41f2b4c87a6c3b0f47c3
I20260924 20:13:19.311269 31746 catalog_manager.cc:4841] TS 6cdd7b4a69ff47bb8ebdfcabd2058cf2 (127.29.83.130:40951): tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 (table test-table [id=af310b61d40e4859b0a3bd204155d570]) successfully deleted
I20260924 20:13:19.311415 32401 ts_tablet_manager.cc:1939] T 67c01015ca3e41f2b4c87a6c3b0f47c3 P 247efd97d0a94f3fa3566d31b000b8d7: Deleting consensus metadata
I20260924 20:13:19.312119 31745 catalog_manager.cc:4841] TS 247efd97d0a94f3fa3566d31b000b8d7 (127.29.83.131:37867): tablet 67c01015ca3e41f2b4c87a6c3b0f47c3 (table test-table [id=af310b61d40e4859b0a3bd204155d570]) successfully deleted
I20260924 20:13:19.315444 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31973
W20260924 20:13:19.315618 31746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:40295 (error 108)
I20260924 20:13:19.317328 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:19.316815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40295 (local address 127.0.0.1:49486)
0924 20:13:19.317188 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:19.317248 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":82,"threads_started":1}
W20260924 20:13:19.317548 31746 catalog_manager.cc:4584] TS d42122042a264557bd70a015a84c173a (127.29.83.129:40295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67c01015ca3e41f2b4c87a6c3b0f47c3: Network error: Client connection negotiation failed: client connection to 127.29.83.129:40295: connect: Connection refused (error 111)
I20260924 20:13:19.322332 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 32113
I20260924 20:13:19.327984 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 31729
2026-09-24T20:13:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6810 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:19 dist-test-slave-52fr krb5kdc[32407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:19 dist-test-slave-52fr krb5kdc[32407](info): set up 2 sockets
Sep 24 20:13:19 dist-test-slave-52fr krb5kdc[32407](info): commencing operation
krb5kdc: starting...
W20260924 20:13:20.369686 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:20 dist-test-slave-52fr krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:20.497884 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45225
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:20.609586 32420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:20.609977 32420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:20.610065 32420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:20.614339 32420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:20.614465 32420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:20.614507 32420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:20.614542 32420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:20.614588 32420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:20.620414 32420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:20.621862 32420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:20.623071 32420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:20.632544 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:20.632539 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:20.632592 32420 server_base.cc:1034] running on GCE node
W20260924 20:13:20.632773 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:20.633337 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:20.634492 32420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:20.635687 32420 hybrid_clock.cc:648] HybridClock initialized: now 1790280800635656 us; error 43 us; skew 500 ppm
Sep 24 20:13:20 dist-test-slave-52fr krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280800, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:20.644050 32420 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:20.645004 32420 webserver.cc:460] Webserver started at http://127.29.83.190:46257/ using document root <none> and password file <none>
I20260924 20:13:20.645589 32420 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:20.645804 32420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:20.646154 32420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:20.648332 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "7246db442ffe4defaa5a5440d4f18f26"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "1bc71b98a19ca6b112acd1789c60d650"
server_key_iv: "787a459617dd706693800e26cad1171a"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.648958 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "7246db442ffe4defaa5a5440d4f18f26"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "1bc71b98a19ca6b112acd1789c60d650"
server_key_iv: "787a459617dd706693800e26cad1171a"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.652678 32420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:13:20.655071 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:20.656257 32420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:20.656451 32420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "7246db442ffe4defaa5a5440d4f18f26"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "1bc71b98a19ca6b112acd1789c60d650"
server_key_iv: "787a459617dd706693800e26cad1171a"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.656646 32420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:20.669957 32420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:20.673216 32420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:20.673436 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:20.680589 32420 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:45225
I20260924 20:13:20.680645 32487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:45225 every 8 connection(s)
I20260924 20:13:20.681577 32420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:20.684535 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 32420
I20260924 20:13:20.684563 32488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:20.684650 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:20.684953 30030 external_mini_cluster.cc:1428] Setting key 31ed31b28bb68c9b3886fb52b64afc7a
I20260924 20:13:20.690992 32488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26: Bootstrap starting.
Sep 24 20:13:20 dist-test-slave-52fr krb5kdc[32407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:20.693627 32488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:20.694641 32488 log.cc:824] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:20.696945 32488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26: No bootstrap required, opened a new log
I20260924 20:13:20.699350 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:20.686383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.29.83.190:45225)
0924 20:13:20.686821 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:20.686836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:20.686870 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:13:20.687535 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:20.687553 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:20.687893 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:20.688223 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:20.689010 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.690060 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.690905 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.691427 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.691597 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:20.694331 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:20.694336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:20.694349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:20.694376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:20.696398 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:20.696935 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:20.696940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:20.696944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:20.697106 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:20.697407 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:20.697410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:20.697411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:20.697897 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:20.698048 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:20.698328 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 20:13:20.698633 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260924 20:13:20.700857 32488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } }
I20260924 20:13:20.701069 32488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:20.701112 32488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7246db442ffe4defaa5a5440d4f18f26, State: Initialized, Role: FOLLOWER
I20260924 20:13:20.701591 32488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } }
I20260924 20:13:20.701718 32488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:20.701835 32488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:20.701951 32488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:20.702857 32488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } }
I20260924 20:13:20.703198 32488 leader_election.cc:302] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7246db442ffe4defaa5a5440d4f18f26; no voters:
I20260924 20:13:20.703549 32488 leader_election.cc:288] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:20.703832 32493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:20.704059 32493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [term 1 LEADER]: Becoming Leader. State: Replica: 7246db442ffe4defaa5a5440d4f18f26, State: Running, Role: LEADER
I20260924 20:13:20.704432 32493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } }
I20260924 20:13:20.704911 32488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:20.706219 32495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7246db442ffe4defaa5a5440d4f18f26. Latest consensus state: current_term: 1 leader_uuid: "7246db442ffe4defaa5a5440d4f18f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } } }
I20260924 20:13:20.706399 32495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:20.706597 32494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7246db442ffe4defaa5a5440d4f18f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7246db442ffe4defaa5a5440d4f18f26" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45225 } } }
I20260924 20:13:20.706954 32494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:20.707094 32502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:20.710698 32502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:20.717250 32502 catalog_manager.cc:1347] Generated new cluster ID: ec6cef3d3d5e47ba8b65dc60e98272a1
I20260924 20:13:20.717345 32502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:20.732882 32502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:20.734229 32502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:20.742165 32502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7246db442ffe4defaa5a5440d4f18f26: Generated new TSK 0
I20260924 20:13:20.742942 32502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:20.776540 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:20.901841 32514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:20.902108 32514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:20.902206 32514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:20.906629 32514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:20.906684 32514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:20.906742 32514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:20.912104 32514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:20.913246 32514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:20.914322 32514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:20.924038 32520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:20.924026 32522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:20.924029 32519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:20.924587 32514 server_base.cc:1034] running on GCE node
I20260924 20:13:20.925117 32514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:20.925858 32514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:20.927037 32514 hybrid_clock.cc:648] HybridClock initialized: now 1790280800927022 us; error 27 us; skew 500 ppm
Sep 24 20:13:20 dist-test-slave-52fr krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280800, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:20.929917 32514 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:20.930819 32514 webserver.cc:460] Webserver started at http://127.29.83.129:44421/ using document root <none> and password file <none>
I20260924 20:13:20.931413 32514 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:20.931509 32514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:20.931723 32514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:20.933810 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "643e73c41a27406780ab4beb9de5ed9d"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "aa3ac58d9ec28e0ea5f82045c601c796"
server_key_iv: "be2204f235e5424b9ecfa8e42160dfea"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.934402 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "643e73c41a27406780ab4beb9de5ed9d"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "aa3ac58d9ec28e0ea5f82045c601c796"
server_key_iv: "be2204f235e5424b9ecfa8e42160dfea"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.938166 32514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:20.940811 32529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:20.942270 32514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:20.942412 32514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "643e73c41a27406780ab4beb9de5ed9d"
format_stamp: "Formatted at 2026-09-24 20:13:20 on dist-test-slave-52fr"
server_key: "aa3ac58d9ec28e0ea5f82045c601c796"
server_key_iv: "be2204f235e5424b9ecfa8e42160dfea"
server_key_version: "encryptionkey@0"
I20260924 20:13:20.942531 32514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:20.956579 32514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:20.959429 32514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:20.959626 32514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:20.960268 32514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:20.961081 32514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:20.961158 32514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:20.961247 32514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:20.961287 32514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:20.969651 32514 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45409
I20260924 20:13:20.969671 32642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45409 every 8 connection(s)
I20260924 20:13:20.970574 32514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:20.973742 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 32514
I20260924 20:13:20.973903 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:20.974161 30030 external_mini_cluster.cc:1428] Setting key 8010efa7b4e8a4248fd20a6fec2bedbc
Sep 24 20:13:20 dist-test-slave-52fr krb5kdc[32407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280800, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:20.984911 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:20.972332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:54459 (local address 127.29.83.190:45225)
0924 20:13:20.972466 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:20.972469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:20.973450 (+ 981us) server_negotiation.cc:407] Connection header received
0924 20:13:20.974472 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:20.974476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:20.974535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:20.974637 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:20.975746 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.976304 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.977827 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.978074 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.978163 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:20.980935 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:20.980942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:20.980947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:20.980981 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:20.982703 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:20.983249 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:20.983252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:20.983254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:20.983350 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:20.983694 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:20.983698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:20.983701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:20.983965 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:20.984087 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:20.984637 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 20:13:20.984783 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:13:20.985809 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:20.972646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45225 (local address 127.29.83.129:54459)
0924 20:13:20.973290 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:20.973330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:20.974273 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:20.974824 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:20.974831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:20.975162 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:20.975613 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:20.975626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.976439 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.976443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:20.977642 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:20.977655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:20.978229 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:20.978236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:20.978395 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:20.979048 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:20.979064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:20.980759 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:20.982864 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:20.982869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:20.982882 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:20.983140 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:20.983450 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:20.983453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:20.983454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:20.983594 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:20.984080 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:20.984085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:20.984319 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 20:13:20.984578 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 20:13:20.984857 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:13:20.987139 32643 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45225
I20260924 20:13:20.987417 32643 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:20.988015 32643 heartbeater.cc:511] Master 127.29.83.190:45225 requested a full tablet report, sending...
I20260924 20:13:20.989665 32452 ts_manager.cc:194] Registered new tserver with Master: 643e73c41a27406780ab4beb9de5ed9d (127.29.83.129:45409)
I20260924 20:13:20.990907 32452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:54459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:21.003017 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:21.111974 32648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:21.112301 32648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:21.112391 32648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:21.116478 32648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:21.116572 32648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:21.116669 32648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:21.121968 32648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:21.123044 32648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:21.124073 32648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:21.133741 32653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:21.133874 32656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:21.133947 32648 server_base.cc:1034] running on GCE node
W20260924 20:13:21.133885 32654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:21.134438 32648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:21.135066 32648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:21.136233 32648 hybrid_clock.cc:648] HybridClock initialized: now 1790280801136213 us; error 30 us; skew 500 ppm
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280801, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:21.139086 32648 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:21.139956 32648 webserver.cc:460] Webserver started at http://127.29.83.130:46357/ using document root <none> and password file <none>
I20260924 20:13:21.140518 32648 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:21.140606 32648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:21.140859 32648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:21.142880 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "a0dd606a27184c3dba0f271b4ac69176"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "e26b447455f51cacf9250e1cb229812f"
server_key_iv: "8b410bd7b96aea177a5706b94bd75743"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.143433 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "a0dd606a27184c3dba0f271b4ac69176"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "e26b447455f51cacf9250e1cb229812f"
server_key_iv: "8b410bd7b96aea177a5706b94bd75743"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.147168 32648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:21.149509 32663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.150724 32648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:21.150851 32648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "a0dd606a27184c3dba0f271b4ac69176"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "e26b447455f51cacf9250e1cb229812f"
server_key_iv: "8b410bd7b96aea177a5706b94bd75743"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.150959 32648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:21.175962 32648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:21.179428 32648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:21.179641 32648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:21.180208 32648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:21.180970 32648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:21.181013 32648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.181053 32648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:21.181069 32648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.189798 32648 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:42801
I20260924 20:13:21.189832 308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:42801 every 8 connection(s)
I20260924 20:13:21.190712 32648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[32407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280801, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:21.200052 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 32648
I20260924 20:13:21.200132 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:21.200433 30030 external_mini_cluster.cc:1428] Setting key c8416e5e7fdf3686d30f24369803ab05
I20260924 20:13:21.204594 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:21.192545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:58047 (local address 127.29.83.190:45225)
0924 20:13:21.192692 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:21.192697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:21.193563 (+ 866us) server_negotiation.cc:407] Connection header received
0924 20:13:21.194926 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:21.194929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:21.194975 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:21.195047 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:21.196211 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.196723 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.197871 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.198093 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.198213 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:21.200755 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:21.200774 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:21.200776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:21.200801 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:21.202412 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:21.203027 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:21.203029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:21.203031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:21.203103 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:21.203489 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:21.203491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:21.203492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:21.203687 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:21.203807 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:21.204287 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 20:13:21.204412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:13:21.205479 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:21.192976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45225 (local address 127.29.83.130:58047)
0924 20:13:21.193412 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:21.193454 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:21.194732 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:21.195213 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:21.195220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:21.195562 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:21.196081 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:21.196095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.196827 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.196829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:21.197712 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:21.197725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.198253 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.198257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:21.198439 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:21.199077 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:21.199092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:21.200595 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:21.202528 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:21.202537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:21.202556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:21.202926 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:21.203189 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:21.203194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:21.203205 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:21.203395 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:21.203805 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:21.203812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:21.204055 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 20:13:21.204243 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 20:13:21.204575 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260924 20:13:21.206915 309 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45225
I20260924 20:13:21.207185 309 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:21.207815 309 heartbeater.cc:511] Master 127.29.83.190:45225 requested a full tablet report, sending...
I20260924 20:13:21.208999 32452 ts_manager.cc:194] Registered new tserver with Master: a0dd606a27184c3dba0f271b4ac69176 (127.29.83.130:42801)
I20260924 20:13:21.209898 32452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:58047
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:21.229094 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:21.340570 314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:21.340848 314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:21.340926 314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:21.344946 314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:21.345032 314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:21.345127 314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:21.350401 314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:21.351491 314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:21.352556 314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:21.362394 320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:21.362406 319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:21.362411 314 server_base.cc:1034] running on GCE node
W20260924 20:13:21.362406 322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:21.363145 314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:21.363729 314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:21.364882 314 hybrid_clock.cc:648] HybridClock initialized: now 1790280801364866 us; error 34 us; skew 500 ppm
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280801, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:21.367596 314 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:21.368371 314 webserver.cc:460] Webserver started at http://127.29.83.131:43583/ using document root <none> and password file <none>
I20260924 20:13:21.368870 314 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:21.368925 314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:21.369079 314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:21.371045 314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "34e030d5dbf143efae83f043bc7a67bd"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "068ae7b021f34d5f0a6bea8597da1f0d"
server_key_iv: "6ce3d91d6ed6c2aa4b32e4310c237e26"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.371547 314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "34e030d5dbf143efae83f043bc7a67bd"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "068ae7b021f34d5f0a6bea8597da1f0d"
server_key_iv: "6ce3d91d6ed6c2aa4b32e4310c237e26"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.375036 314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:21.377303 329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.378530 314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:21.378669 314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "34e030d5dbf143efae83f043bc7a67bd"
format_stamp: "Formatted at 2026-09-24 20:13:21 on dist-test-slave-52fr"
server_key: "068ae7b021f34d5f0a6bea8597da1f0d"
server_key_iv: "6ce3d91d6ed6c2aa4b32e4310c237e26"
server_key_version: "encryptionkey@0"
I20260924 20:13:21.378780 314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:21.388417 314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:21.391474 314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:21.391692 314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:21.392282 314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:21.393085 314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:21.393153 314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.393249 314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:21.393299 314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:21.402496 314 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:45165
I20260924 20:13:21.402560 442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:45165 every 8 connection(s)
I20260924 20:13:21.403504 314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:21.405284 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 314
I20260924 20:13:21.405417 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:21.405834 30030 external_mini_cluster.cc:1428] Setting key 2ca0cd9a0bd967752041c0afbdf03527
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[32407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280801, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:21.418433 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:21.405573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:59437 (local address 127.29.83.190:45225)
0924 20:13:21.405949 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:21.405954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:21.407082 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 20:13:21.408503 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:21.408506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:21.408563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:21.408636 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:21.409931 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.410481 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.411626 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.411871 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.411985 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:21.414718 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:21.414721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:21.414723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:21.414750 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:21.416169 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:21.416757 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:21.416761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:21.416763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:21.416843 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:21.417201 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:21.417204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:21.417205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:21.417394 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:21.417470 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:21.418108 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 20:13:21.418237 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260924 20:13:21.419462 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:21.406189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45225 (local address 127.29.83.131:59437)
0924 20:13:21.406922 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:21.406980 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:21.408284 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:21.408818 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:21.408826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:21.409199 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:21.409725 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:21.409738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.410610 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.410614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:21.411502 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:21.411512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:21.412030 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:21.412034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:21.412211 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:21.412849 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:21.412867 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:21.414519 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:21.416293 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:21.416299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:21.416313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:21.416653 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:21.416952 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:21.416955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:21.416956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:21.417107 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:21.417532 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:21.417538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:21.417838 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 20:13:21.418022 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 20:13:21.418332 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":153,"threads_started":1}
I20260924 20:13:21.420558 443 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45225
I20260924 20:13:21.420840 443 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:21.421341 443 heartbeater.cc:511] Master 127.29.83.190:45225 requested a full tablet report, sending...
I20260924 20:13:21.422394 32452 ts_manager.cc:194] Registered new tserver with Master: 34e030d5dbf143efae83f043bc7a67bd (127.29.83.131:45165)
I20260924 20:13:21.422919 32452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:59437
I20260924 20:13:21.431667 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:13:21.444424 32491 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:21.442997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.29.83.190:45225)
0924 20:13:21.443128 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:21.443132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:21.443328 (+ 196us) server_negotiation.cc:407] Connection header received
0924 20:13:21.443433 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:21.443436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:21.443483 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:21.443566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:21.444252 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42096: BlockingRecv error: recv got EOF from 127.0.0.1:42096 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:21.445611 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 32514
I20260924 20:13:21.451069 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 32648
I20260924 20:13:21.456642 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 314
I20260924 20:13:21.462026 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 32420
2026-09-24T20:13:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2132 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[455](info): set up 2 sockets
Sep 24 20:13:21 dist-test-slave-52fr krb5kdc[455](info): commencing operation
krb5kdc: starting...
W20260924 20:13:22.502033 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:22.622985 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41047
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:22.733067 468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:22.733332 468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:22.733373 468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:22.737449 468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:22.737527 468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:22.737547 468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:22.737569 468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:22.737586 468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:22.743005 468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:22.744149 468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:22.745172 468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:22.753434 468 server_base.cc:1034] running on GCE node
W20260924 20:13:22.753369 476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:22.753363 474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:22.753355 473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:22.754282 468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:22.755331 468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:22.756490 468 hybrid_clock.cc:648] HybridClock initialized: now 1790280802756474 us; error 30 us; skew 500 ppm
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280802, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:22.759240 468 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:22.760041 468 webserver.cc:460] Webserver started at http://127.29.83.190:33213/ using document root <none> and password file <none>
I20260924 20:13:22.760587 468 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:22.760675 468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:22.760901 468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:22.763041 468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "6b092b2d9bff4fdf89d9af0301383a9f"
format_stamp: "Formatted at 2026-09-24 20:13:22 on dist-test-slave-52fr"
server_key: "bfa87ff5ebce699ef7dc464958986693"
server_key_iv: "02a480e2a60446b63e43b598538eff04"
server_key_version: "encryptionkey@0"
I20260924 20:13:22.763603 468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "6b092b2d9bff4fdf89d9af0301383a9f"
format_stamp: "Formatted at 2026-09-24 20:13:22 on dist-test-slave-52fr"
server_key: "bfa87ff5ebce699ef7dc464958986693"
server_key_iv: "02a480e2a60446b63e43b598538eff04"
server_key_version: "encryptionkey@0"
I20260924 20:13:22.767162 468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:22.769507 483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:22.770781 468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:22.770921 468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "6b092b2d9bff4fdf89d9af0301383a9f"
format_stamp: "Formatted at 2026-09-24 20:13:22 on dist-test-slave-52fr"
server_key: "bfa87ff5ebce699ef7dc464958986693"
server_key_iv: "02a480e2a60446b63e43b598538eff04"
server_key_version: "encryptionkey@0"
I20260924 20:13:22.771034 468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:22.804962 468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:22.807969 468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:22.808188 468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:22.815235 468 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:41047
I20260924 20:13:22.815527 535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:41047 every 8 connection(s)
I20260924 20:13:22.816337 468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:22.819180 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:22.819713 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 468
I20260924 20:13:22.819830 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:22.820129 30030 external_mini_cluster.cc:1428] Setting key 958255dfc1e443b4ddf66c6372b24cb9
I20260924 20:13:22.826063 536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f: Bootstrap starting.
I20260924 20:13:22.828397 536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f: Neither blocks nor log segments found. Creating new log.
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:22.829324 536 log.cc:824] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:22.831260 536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f: No bootstrap required, opened a new log
I20260924 20:13:22.834497 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:22.821622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.29.83.190:41047)
0924 20:13:22.822105 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:22.822119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:22.822154 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:13:22.822824 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:22.822842 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:22.823180 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:22.823554 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:22.824419 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:22.825392 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:22.826386 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:22.826862 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:22.827034 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:22.829842 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:22.829847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:22.829860 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:22.829887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:22.831658 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:22.832103 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:22.832109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:22.832114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:22.832222 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:22.832504 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:22.832508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:22.832510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:22.832911 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:22.833108 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:22.833413 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 20:13:22.833702 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":165,"threads_started":1}
I20260924 20:13:22.834996 536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } }
I20260924 20:13:22.835268 536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:22.835350 536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b092b2d9bff4fdf89d9af0301383a9f, State: Initialized, Role: FOLLOWER
I20260924 20:13:22.835888 536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } }
I20260924 20:13:22.836050 536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:22.836112 536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:22.836208 536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:22.837407 536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } }
I20260924 20:13:22.837617 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:13:22.837810 536 leader_election.cc:302] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b092b2d9bff4fdf89d9af0301383a9f; no voters:
I20260924 20:13:22.838187 536 leader_election.cc:288] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:22.838260 541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:22.838477 541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [term 1 LEADER]: Becoming Leader. State: Replica: 6b092b2d9bff4fdf89d9af0301383a9f, State: Running, Role: LEADER
I20260924 20:13:22.838822 541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } }
I20260924 20:13:22.839494 536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:22.840471 543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b092b2d9bff4fdf89d9af0301383a9f. Latest consensus state: current_term: 1 leader_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } } }
I20260924 20:13:22.840602 543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:22.840996 542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b092b2d9bff4fdf89d9af0301383a9f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41047 } } }
I20260924 20:13:22.841099 542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:22.841495 551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:22.844252 551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:22.851119 551 catalog_manager.cc:1347] Generated new cluster ID: a5eaf11583e84b00bc86b92351f98b70
I20260924 20:13:22.851222 551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:22.863271 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:22.857612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.29.83.190:41047)
0924 20:13:22.857745 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:22.857750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:22.858000 (+ 250us) server_negotiation.cc:407] Connection header received
0924 20:13:22.858101 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:22.858105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:22.858156 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:22.858264 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:22.858813 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:22.859640 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:22.860747 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:22.861115 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:22.861227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:22.862158 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:22.862162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:22.862165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:22.862791 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:22.862827 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:22.862849 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:22.862945 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:22.863045 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 20:13:22.863108 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":43}
I20260924 20:13:22.866585 551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:22.867434 551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:22.884377 551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b092b2d9bff4fdf89d9af0301383a9f: Generated new TSK 0
I20260924 20:13:22.885342 551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:13:22.897274 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 468
2026-09-24T20:13:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1433 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[569](info): set up 2 sockets
Sep 24 20:13:22 dist-test-slave-52fr krb5kdc[569](info): commencing operation
krb5kdc: starting...
W20260924 20:13:23.949926 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:24.075219 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35593
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:24.186384 582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:24.186725 582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:24.186813 582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:24.190953 582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:24.191061 582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:24.191099 582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:24.191138 582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:24.191172 582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:24.196630 582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:35593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:24.197829 582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:24.198917 582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:24.207934 582 server_base.cc:1034] running on GCE node
W20260924 20:13:24.207894 590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.207901 588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.207894 587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:24.208658 582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:24.209719 582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:24.210999 582 hybrid_clock.cc:648] HybridClock initialized: now 1790280804210984 us; error 27 us; skew 500 ppm
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:24.213934 582 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:24.214823 582 webserver.cc:460] Webserver started at http://127.29.83.190:44469/ using document root <none> and password file <none>
I20260924 20:13:24.215348 582 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:24.215404 582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:24.215566 582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:24.217574 582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "8e65563bfda14e8d8c9e78c96eb1280e"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "a5977cfbeb21318c14c14e82dd0bbcb9"
server_key_iv: "563a856ae06d4f498d37c7d7cc17f8a1"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.218166 582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "8e65563bfda14e8d8c9e78c96eb1280e"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "a5977cfbeb21318c14c14e82dd0bbcb9"
server_key_iv: "563a856ae06d4f498d37c7d7cc17f8a1"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.221952 582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:24.224345 597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.225652 582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:24.225844 582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "8e65563bfda14e8d8c9e78c96eb1280e"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "a5977cfbeb21318c14c14e82dd0bbcb9"
server_key_iv: "563a856ae06d4f498d37c7d7cc17f8a1"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.225973 582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:24.265883 582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:24.269269 582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:24.269488 582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:24.276377 582 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:35593
I20260924 20:13:24.276661 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:35593 every 8 connection(s)
I20260924 20:13:24.277352 582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:24.280059 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:24.282125 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 582
I20260924 20:13:24.282272 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:24.282557 30030 external_mini_cluster.cc:1428] Setting key 8fbd56d1c10b1ba63eeb64a8f7219693
I20260924 20:13:24.286487 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e: Bootstrap starting.
I20260924 20:13:24.288626 650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:24.289372 650 log.cc:824] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:24.291429 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e: No bootstrap required, opened a new log
I20260924 20:13:24.294960 650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } }
I20260924 20:13:24.295202 650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:24.295270 650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e65563bfda14e8d8c9e78c96eb1280e, State: Initialized, Role: FOLLOWER
I20260924 20:13:24.295676 650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } }
I20260924 20:13:24.295841 650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:24.295923 650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:24.296046 650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:24.296343 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:24.283871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.29.83.190:35593)
0924 20:13:24.284286 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:24.284300 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:24.284334 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:13:24.284972 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:24.284988 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:24.285302 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:24.285624 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:24.286464 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.287462 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.288332 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.288848 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.289022 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:24.291721 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:24.291725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:24.291737 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:24.291764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:24.293383 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.293853 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.293858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.293863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.293963 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.294239 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.294242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.294244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.294611 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:24.294822 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:24.295109 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 20:13:24.295421 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260924 20:13:24.297024 650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } }
I20260924 20:13:24.297369 650 leader_election.cc:302] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e65563bfda14e8d8c9e78c96eb1280e; no voters:
I20260924 20:13:24.297709 650 leader_election.cc:288] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:24.297816 655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:24.297981 655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [term 1 LEADER]: Becoming Leader. State: Replica: 8e65563bfda14e8d8c9e78c96eb1280e, State: Running, Role: LEADER
I20260924 20:13:24.298353 655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } }
I20260924 20:13:24.299028 650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:24.300172 656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } } }
I20260924 20:13:24.300313 656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:24.300587 657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e65563bfda14e8d8c9e78c96eb1280e. Latest consensus state: current_term: 1 leader_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e65563bfda14e8d8c9e78c96eb1280e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35593 } } }
I20260924 20:13:24.300669 664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:24.300745 657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:24.303977 664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:24.309731 664 catalog_manager.cc:1347] Generated new cluster ID: dda72454b0534ad0a6fca03ebef6ceff
I20260924 20:13:24.309818 664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:24.334040 664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:24.335404 664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:24.341887 664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e65563bfda14e8d8c9e78c96eb1280e: Generated new TSK 0
I20260924 20:13:24.342659 664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:24.371088 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:24.481984 676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:24.482256 676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:24.482332 676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:24.487187 676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:24.487282 676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:24.487367 676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:24.494055 676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:24.495214 676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:24.496331 676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:24.506430 682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.506433 681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.506430 684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:24.506592 676 server_base.cc:1034] running on GCE node
I20260924 20:13:24.507087 676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:24.507754 676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:24.508919 676 hybrid_clock.cc:648] HybridClock initialized: now 1790280804508884 us; error 47 us; skew 500 ppm
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:24.511807 676 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:24.512705 676 webserver.cc:460] Webserver started at http://127.29.83.129:39877/ using document root <none> and password file <none>
I20260924 20:13:24.513273 676 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:24.513360 676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:24.513593 676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:24.515791 676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "5455e9b1a4a44741867dfa1774aa3759"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "54d079fbbf7ff25d7447bca57b89e8f6"
server_key_iv: "e8d7872bb6d136d38b64fde0fa712ca7"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.516368 676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "5455e9b1a4a44741867dfa1774aa3759"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "54d079fbbf7ff25d7447bca57b89e8f6"
server_key_iv: "e8d7872bb6d136d38b64fde0fa712ca7"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.520330 676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 20:13:24.523082 691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.524410 676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:24.524554 676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "5455e9b1a4a44741867dfa1774aa3759"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "54d079fbbf7ff25d7447bca57b89e8f6"
server_key_iv: "e8d7872bb6d136d38b64fde0fa712ca7"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.524669 676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:24.553239 676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:24.556929 676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:24.557149 676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:24.557857 676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:24.558719 676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:24.558766 676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.558807 676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:24.558859 676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.572722 805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:44181 every 8 connection(s)
I20260924 20:13:24.572737 676 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:44181
I20260924 20:13:24.574231 676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:24.578441 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 676
I20260924 20:13:24.578606 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:24.578905 30030 external_mini_cluster.cc:1428] Setting key 7efa53d19555d8775e6d968f51a3c2dc
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:13:24.594393 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:24.577471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:54325 (local address 127.29.83.190:35593)
0924 20:13:24.577632 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:24.577637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:24.579032 (+ 1395us) server_negotiation.cc:407] Connection header received
0924 20:13:24.580795 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:24.580799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:24.580852 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:24.580930 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:24.582225 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.583125 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.585024 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.585476 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.585607 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:24.589371 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:24.589375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:24.589378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:24.589414 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:24.591394 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.592132 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.592135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.592137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.592238 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.592652 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.592656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.592657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.593009 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:24.593170 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:24.593964 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 20:13:24.594124 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:13:24.595533 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:24.577996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35593 (local address 127.29.83.129:54325)
0924 20:13:24.578847 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:24.578916 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:24.580574 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:24.581123 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:24.581130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:24.581497 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:24.582073 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:24.582088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.583304 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.583309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:24.584876 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:24.584891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.585679 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.585686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:24.585946 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:24.587029 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:24.587058 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:24.589160 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:24.591564 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:24.591573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:24.591594 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:24.592027 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:24.592328 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:24.592331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:24.592333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:24.592556 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:24.593241 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:24.593251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:24.593611 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 20:13:24.593882 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 20:13:24.594286 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":183,"threads_started":1}
I20260924 20:13:24.596917 806 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35593
I20260924 20:13:24.597199 806 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:24.597910 806 heartbeater.cc:511] Master 127.29.83.190:35593 requested a full tablet report, sending...
I20260924 20:13:24.599931 614 ts_manager.cc:194] Registered new tserver with Master: 5455e9b1a4a44741867dfa1774aa3759 (127.29.83.129:44181)
I20260924 20:13:24.601593 614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:54325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:24.611256 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:24.725242 811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:24.725500 811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:24.725597 811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:24.730068 811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:24.730127 811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:24.730213 811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:24.735569 811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:24.736639 811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:24.737732 811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:24.747488 817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.747503 816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.747514 819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:24.747555 811 server_base.cc:1034] running on GCE node
I20260924 20:13:24.748171 811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:24.748813 811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:24.749974 811 hybrid_clock.cc:648] HybridClock initialized: now 1790280804749956 us; error 27 us; skew 500 ppm
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:24.752772 811 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:24.753638 811 webserver.cc:460] Webserver started at http://127.29.83.130:39133/ using document root <none> and password file <none>
I20260924 20:13:24.754233 811 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:24.754293 811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:24.754520 811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:24.756486 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "6dffd7a271e74b5dadf562f2c08fb174"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "47e6df7329aec19a14fcf66021f2c1d7"
server_key_iv: "0b67e171c06f563286afaf4ba97a11fc"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.757030 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "6dffd7a271e74b5dadf562f2c08fb174"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "47e6df7329aec19a14fcf66021f2c1d7"
server_key_iv: "0b67e171c06f563286afaf4ba97a11fc"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.760672 811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:13:24.763149 826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.764456 811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:24.764626 811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "6dffd7a271e74b5dadf562f2c08fb174"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "47e6df7329aec19a14fcf66021f2c1d7"
server_key_iv: "0b67e171c06f563286afaf4ba97a11fc"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.764741 811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:24.774675 811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:24.777976 811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:24.778174 811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:24.778738 811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:24.779520 811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:24.779589 811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.779673 811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:24.779723 811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.788270 811 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:46163
I20260924 20:13:24.788292 939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:46163 every 8 connection(s)
I20260924 20:13:24.789134 811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:24.798583 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 811
I20260924 20:13:24.798704 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:24.799045 30030 external_mini_cluster.cc:1428] Setting key 6dccf5590384ebb03ed6dc4a0bd8ebfd
I20260924 20:13:24.803301 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:24.791137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:47775 (local address 127.29.83.190:35593)
0924 20:13:24.791272 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:24.791275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:24.792035 (+ 760us) server_negotiation.cc:407] Connection header received
0924 20:13:24.793406 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:24.793409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:24.793462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:24.793549 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:24.794761 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.795289 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.796580 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.796814 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.796925 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:24.799513 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:24.799516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:24.799518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:24.799541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:24.801269 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.801804 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.801808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.801810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.801892 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:24.802238 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:24.802241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:24.802243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:24.802422 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:24.802525 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:24.803024 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 20:13:24.803138 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:13:24.804136 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:24.791460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35593 (local address 127.29.83.130:47775)
0924 20:13:24.791890 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:24.791930 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:24.793185 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:24.793738 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:24.793744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:24.794142 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:24.794614 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:24.794627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.795415 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.795421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:24.796455 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:24.796465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:24.796964 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:24.796968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:24.797135 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:24.797697 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:24.797712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:24.799344 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:24.801387 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:24.801393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:24.801408 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:24.801665 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:24.801984 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:24.801987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:24.801989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:24.802130 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:24.802523 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:24.802530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:24.802784 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:13:24.802983 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 20:13:24.803317 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260924 20:13:24.805356 940 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35593
I20260924 20:13:24.805629 940 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:24.806291 940 heartbeater.cc:511] Master 127.29.83.190:35593 requested a full tablet report, sending...
I20260924 20:13:24.807523 614 ts_manager.cc:194] Registered new tserver with Master: 6dffd7a271e74b5dadf562f2c08fb174 (127.29.83.130:46163)
I20260924 20:13:24.808161 614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:47775
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:24.826683 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:24.940968 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:24.941254 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:24.941300 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:24.945672 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:24.945739 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:24.945874 945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:24.951241 945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:24.952252 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:24.953212 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:24.962746 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:24.962796 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:24.962783 945 server_base.cc:1034] running on GCE node
W20260924 20:13:24.963186 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:24.963559 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:24.964160 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:24.965339 945 hybrid_clock.cc:648] HybridClock initialized: now 1790280804965324 us; error 27 us; skew 500 ppm
Sep 24 20:13:24 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:24.968067 945 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:24.969009 945 webserver.cc:460] Webserver started at http://127.29.83.131:46545/ using document root <none> and password file <none>
I20260924 20:13:24.969555 945 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:24.969612 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:24.969895 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:24.971879 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "3830cda3384e458599950a861ced565d"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "6a6d243aa510030c42db295d5fbfc5f6"
server_key_iv: "0525a79ab9ce0f28317901f15c93d0c0"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.972427 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "3830cda3384e458599950a861ced565d"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "6a6d243aa510030c42db295d5fbfc5f6"
server_key_iv: "0525a79ab9ce0f28317901f15c93d0c0"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.975989 945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:13:24.978502 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:24.979682 945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:13:24.979776 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "3830cda3384e458599950a861ced565d"
format_stamp: "Formatted at 2026-09-24 20:13:24 on dist-test-slave-52fr"
server_key: "6a6d243aa510030c42db295d5fbfc5f6"
server_key_iv: "0525a79ab9ce0f28317901f15c93d0c0"
server_key_version: "encryptionkey@0"
I20260924 20:13:24.979904 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:25.000169 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:25.003553 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:25.003808 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:25.004494 945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:25.005375 945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:25.005422 945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:25.005465 945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:25.005479 945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:25.014854 945 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:39367
I20260924 20:13:25.014875 1073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:39367 every 8 connection(s)
I20260924 20:13:25.015841 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:25.023993 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 945
I20260924 20:13:25.024132 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:25.024425 30030 external_mini_cluster.cc:1428] Setting key 40470e108f3a292668f103777595efdc
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:25.029898 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:25.017822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:54223 (local address 127.29.83.190:35593)
0924 20:13:25.017975 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:25.017979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:25.018812 (+ 833us) server_negotiation.cc:407] Connection header received
0924 20:13:25.020217 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:25.020220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:25.020269 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:25.020345 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:25.021526 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.022141 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.023235 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.023446 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.023529 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:25.026135 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:25.026138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:25.026141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:25.026177 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:25.027664 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:25.028189 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:25.028195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:25.028199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:25.028286 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:25.028711 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:25.028715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:25.028716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:25.028909 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:25.029014 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:25.029531 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 20:13:25.029695 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:13:25.030682 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:25.018211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35593 (local address 127.29.83.131:54223)
0924 20:13:25.018643 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:25.018688 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:25.019989 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:25.020513 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:25.020520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:25.020882 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:25.021378 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:25.021394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.022264 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.022267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:25.023126 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:25.023134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.023574 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.023577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:25.023724 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:25.024334 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:25.024349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:25.025975 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:25.027798 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:25.027805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:25.027821 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:25.028085 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:25.028384 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:25.028387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:25.028389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:25.028594 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:25.029023 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:25.029032 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:25.029302 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 20:13:25.029494 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:13:25.029853 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260924 20:13:25.031800 1074 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35593
I20260924 20:13:25.032058 1074 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:25.032565 1074 heartbeater.cc:511] Master 127.29.83.190:35593 requested a full tablet report, sending...
I20260924 20:13:25.033650 614 ts_manager.cc:194] Registered new tserver with Master: 3830cda3384e458599950a861ced565d (127.29.83.131:39367)
I20260924 20:13:25.034382 614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:54223
I20260924 20:13:25.038632 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:25.064236 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:25.055914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.29.83.190:35593)
0924 20:13:25.056103 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:25.056107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:25.056166 (+ 59us) server_negotiation.cc:407] Connection header received
0924 20:13:25.056296 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:25.056299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:25.056341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:25.056411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:25.057071 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.057582 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.058450 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:25.058658 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:25.058736 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:25.061041 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:25.061045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:25.061047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:25.061071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:25.062390 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:25.062812 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:25.062815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:25.062817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:25.062907 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:25.063264 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:25.063268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:25.063269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:25.063503 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:25.063634 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:25.063991 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 20:13:25.064103 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 20:13:25.064814 614 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52786
I20260924 20:13:25.065830 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 676
I20260924 20:13:25.071561 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 811
I20260924 20:13:25.076889 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 945
I20260924 20:13:25.082305 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 582
2026-09-24T20:13:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2187 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[1086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[1086](info): set up 2 sockets
Sep 24 20:13:25 dist-test-slave-52fr krb5kdc[1086](info): commencing operation
krb5kdc: starting...
W20260924 20:13:26.117848 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:26.239230 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46229
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:26.350199 1099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:26.350543 1099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:26.350626 1099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:26.354833 1099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:26.354923 1099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:26.354964 1099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:26.355022 1099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:26.355068 1099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:26.360548 1099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:26.361665 1099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:26.362767 1099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:26.371573 1104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:26.371562 1107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:26.371734 1105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:26.371876 1099 server_base.cc:1034] running on GCE node
I20260924 20:13:26.372606 1099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:26.373809 1099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:26.375032 1099 hybrid_clock.cc:648] HybridClock initialized: now 1790280806375015 us; error 28 us; skew 500 ppm
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:26.384244 1099 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:26.385283 1099 webserver.cc:460] Webserver started at http://127.29.83.190:40275/ using document root <none> and password file <none>
I20260924 20:13:26.385924 1099 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:26.386014 1099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:26.386245 1099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:26.388298 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "5107310659f64d1da77c02b4f6218075"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "a066c1b355f1131409d66bc2f5b5ae6b"
server_key_iv: "f6f8f454f113b2e4f13f19b37d1ff1ee"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.388861 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "5107310659f64d1da77c02b4f6218075"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "a066c1b355f1131409d66bc2f5b5ae6b"
server_key_iv: "f6f8f454f113b2e4f13f19b37d1ff1ee"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.392653 1099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:13:26.395102 1114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.396404 1099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:26.396538 1099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5107310659f64d1da77c02b4f6218075"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "a066c1b355f1131409d66bc2f5b5ae6b"
server_key_iv: "f6f8f454f113b2e4f13f19b37d1ff1ee"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.396651 1099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:26.408353 1099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:26.411890 1099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:26.412101 1099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:26.419062 1099 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:46229
I20260924 20:13:26.419143 1166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:46229 every 8 connection(s)
I20260924 20:13:26.420061 1099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:26.422900 1167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:26.425559 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1099
I20260924 20:13:26.425741 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:26.426038 30030 external_mini_cluster.cc:1428] Setting key 8a4ceb997fdb393e23fc41e8df9f8441
I20260924 20:13:26.428786 1167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: Bootstrap starting.
I20260924 20:13:26.431434 1167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:26.432240 1167 log.cc:824] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:26.434379 1167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: No bootstrap required, opened a new log
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:26.437866 1167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } }
I20260924 20:13:26.438094 1167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:26.438175 1167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5107310659f64d1da77c02b4f6218075, State: Initialized, Role: FOLLOWER
I20260924 20:13:26.438617 1167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } }
I20260924 20:13:26.438738 1167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:26.438824 1167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:26.438928 1167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:26.439846 1167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } }
I20260924 20:13:26.440201 1167 leader_election.cc:302] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5107310659f64d1da77c02b4f6218075; no voters:
I20260924 20:13:26.440553 1167 leader_election.cc:288] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:26.440701 1172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:26.440938 1172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [term 1 LEADER]: Becoming Leader. State: Replica: 5107310659f64d1da77c02b4f6218075, State: Running, Role: LEADER
I20260924 20:13:26.441241 1172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } }
I20260924 20:13:26.441877 1167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:26.441977 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:26.427416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.29.83.190:46229)
0924 20:13:26.427884 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:26.427904 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:26.427952 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:13:26.428727 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:26.428750 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:26.429157 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:26.429518 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:26.430262 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.431743 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.432649 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.433261 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.433426 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:26.436234 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:26.436239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:26.436254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:26.436282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:26.438768 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.439303 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.439309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.439315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.439440 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.439727 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.439730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.439732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.440208 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:26.440414 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:26.440706 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 20:13:26.441071 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":145,"threads_started":1}
I20260924 20:13:26.442448 1172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5107310659f64d1da77c02b4f6218075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } } }
I20260924 20:13:26.442560 1172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:26.442771 1172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5107310659f64d1da77c02b4f6218075. Latest consensus state: current_term: 1 leader_uuid: "5107310659f64d1da77c02b4f6218075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5107310659f64d1da77c02b4f6218075" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46229 } } }
I20260924 20:13:26.442870 1172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075 [sys.catalog]: This master's current role is: LEADER
W20260924 20:13:26.446516 1187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:26.446578 1187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:26.446635 1178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:26.447337 1178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:26.452301 1178 catalog_manager.cc:1347] Generated new cluster ID: c734c9c6e4da4fd4875e1cf7fc6c3c94
I20260924 20:13:26.452373 1178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:26.466498 1178 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:26.467854 1178 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:26.483279 1178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5107310659f64d1da77c02b4f6218075: Generated new TSK 0
I20260924 20:13:26.484184 1178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:26.524816 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:26.639809 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:26.640156 1193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:26.640230 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:26.644783 1193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:26.644874 1193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:26.644971 1193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:26.650446 1193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:26.651554 1193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:26.652621 1193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:26.662408 1199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:26.662428 1198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:26.662479 1193 server_base.cc:1034] running on GCE node
W20260924 20:13:26.662423 1201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:26.662954 1193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:26.663636 1193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:26.664865 1193 hybrid_clock.cc:648] HybridClock initialized: now 1790280806664846 us; error 36 us; skew 500 ppm
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:26.667686 1193 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:26.668476 1193 webserver.cc:460] Webserver started at http://127.29.83.129:32885/ using document root <none> and password file <none>
I20260924 20:13:26.668974 1193 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:26.669031 1193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:26.669198 1193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:26.671235 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "4b03e173bfd84e0ba65ef3a19a94b8c3"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "28ee52fb45b6061d1adce4ad481091cd"
server_key_iv: "fb29bb99da5d04a0d8cf6826d3955905"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.671736 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "4b03e173bfd84e0ba65ef3a19a94b8c3"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "28ee52fb45b6061d1adce4ad481091cd"
server_key_iv: "fb29bb99da5d04a0d8cf6826d3955905"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.675344 1193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:13:26.677734 1208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.679028 1193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:26.679173 1193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "4b03e173bfd84e0ba65ef3a19a94b8c3"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "28ee52fb45b6061d1adce4ad481091cd"
server_key_iv: "fb29bb99da5d04a0d8cf6826d3955905"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.679286 1193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:26.705982 1193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:26.709174 1193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:26.709414 1193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:26.710080 1193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:26.710932 1193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:26.710979 1193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.711050 1193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:26.711081 1193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.719949 1193 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:41195
I20260924 20:13:26.719966 1321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:41195 every 8 connection(s)
I20260924 20:13:26.720870 1193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:26.721905 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1193
I20260924 20:13:26.722035 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:26.722364 30030 external_mini_cluster.cc:1428] Setting key 02c478d16f9c2c3730f6ce87623abbe7
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:26.739151 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:26.722984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:47481 (local address 127.29.83.190:46229)
0924 20:13:26.723150 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:26.723155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:26.724151 (+ 996us) server_negotiation.cc:407] Connection header received
0924 20:13:26.725738 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:26.725741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:26.725834 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:26.725913 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:26.727359 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.728285 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.729518 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.729936 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.730059 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:26.735177 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:26.735181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:26.735183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:26.735209 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:26.736883 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.737385 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.737391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.737396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.737493 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.737861 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.737865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.737868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.738144 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:26.738268 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:26.738744 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 20:13:26.738901 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:13:26.739907 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:26.723476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46229 (local address 127.29.83.129:47481)
0924 20:13:26.723976 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:26.724037 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:26.725491 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:26.726113 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:26.726124 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:26.726670 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:26.727205 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:26.727219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.728423 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.728426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:26.729383 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:26.729394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.730106 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.730109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:26.730279 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:26.730981 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:26.730999 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:26.734958 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:26.736998 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:26.737003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:26.737016 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:26.737283 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:26.737585 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:26.737587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:26.737589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:26.737725 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:26.738256 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:26.738261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:26.738501 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 20:13:26.738657 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 20:13:26.738952 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":149,"threads_started":1}
I20260924 20:13:26.741158 1322 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46229
I20260924 20:13:26.741397 1322 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:26.742013 1322 heartbeater.cc:511] Master 127.29.83.190:46229 requested a full tablet report, sending...
I20260924 20:13:26.743774 1131 ts_manager.cc:194] Registered new tserver with Master: 4b03e173bfd84e0ba65ef3a19a94b8c3 (127.29.83.129:41195)
I20260924 20:13:26.745005 1131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:47481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:26.750926 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:26.861290 1327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:26.861598 1327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:26.861682 1327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:26.865728 1327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:26.865854 1327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:26.865952 1327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:26.871330 1327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:26.872481 1327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:26.873543 1327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:26.883600 1332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:26.883502 1333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:26.883486 1335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:26.883558 1327 server_base.cc:1034] running on GCE node
I20260924 20:13:26.884210 1327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:26.884852 1327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:26.886016 1327 hybrid_clock.cc:648] HybridClock initialized: now 1790280806885974 us; error 52 us; skew 500 ppm
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:26.888756 1327 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:26.889685 1327 webserver.cc:460] Webserver started at http://127.29.83.130:37823/ using document root <none> and password file <none>
I20260924 20:13:26.890264 1327 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:26.890347 1327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:26.890571 1327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:26.892604 1327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "fa17160d61254552861e960de19cbb4c"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "f515454f2ddc8cf19829d610747b03c7"
server_key_iv: "3c5fcb6a4c5267e4c8f3dd028ecb47dd"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.893163 1327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "fa17160d61254552861e960de19cbb4c"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "f515454f2ddc8cf19829d610747b03c7"
server_key_iv: "3c5fcb6a4c5267e4c8f3dd028ecb47dd"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.896910 1327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:13:26.899370 1342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.900561 1327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:26.900717 1327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "fa17160d61254552861e960de19cbb4c"
format_stamp: "Formatted at 2026-09-24 20:13:26 on dist-test-slave-52fr"
server_key: "f515454f2ddc8cf19829d610747b03c7"
server_key_iv: "3c5fcb6a4c5267e4c8f3dd028ecb47dd"
server_key_version: "encryptionkey@0"
I20260924 20:13:26.900827 1327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:26.912546 1327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:26.915448 1327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:26.915674 1327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:26.916258 1327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:26.917104 1327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:26.917177 1327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.917248 1327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:26.917300 1327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:26.926508 1327 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:37049
I20260924 20:13:26.926544 1455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:37049 every 8 connection(s)
I20260924 20:13:26.927416 1327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:13:26 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:26.937376 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1327
I20260924 20:13:26.937481 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:26.937820 30030 external_mini_cluster.cc:1428] Setting key df3f6f6507f6a6dbb203fc3a5e5129ed
I20260924 20:13:26.941184 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:26.929177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:58107 (local address 127.29.83.190:46229)
0924 20:13:26.929325 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:26.929329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:26.930214 (+ 885us) server_negotiation.cc:407] Connection header received
0924 20:13:26.931573 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:26.931576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:26.931634 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:26.931744 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:26.932883 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.933414 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.934584 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.934808 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.934916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:26.937340 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:26.937343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:26.937345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:26.937369 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:26.939023 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.939601 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.939606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.939608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.939726 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:26.940034 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:26.940040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:26.940045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:26.940288 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:26.940392 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:26.940847 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 20:13:26.940986 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:13:26.942337 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:26.929569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46229 (local address 127.29.83.130:58107)
0924 20:13:26.930062 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:26.930103 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:26.931370 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:26.931930 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:26.931937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:26.932275 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:26.932737 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:26.932751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.933536 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.933540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:26.934462 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:26.934472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:26.934952 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:26.934956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:26.935118 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:26.935707 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:26.935723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:26.937142 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:26.939139 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:26.939145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:26.939159 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:26.939493 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:26.939819 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:26.939821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:26.939823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:26.939944 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:26.940389 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:26.940394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:26.940608 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 20:13:26.940799 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:13:26.941191 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":172,"threads_started":1}
I20260924 20:13:26.943872 1456 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46229
I20260924 20:13:26.944131 1456 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:26.944628 1456 heartbeater.cc:511] Master 127.29.83.190:46229 requested a full tablet report, sending...
I20260924 20:13:26.945649 1131 ts_manager.cc:194] Registered new tserver with Master: fa17160d61254552861e960de19cbb4c (127.29.83.130:37049)
I20260924 20:13:26.946244 1131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:58107
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:26.964390 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:27.074295 1461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:27.074582 1461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:27.074627 1461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:27.078732 1461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:27.078806 1461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:27.078866 1461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:27.084429 1461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:27.085595 1461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:27.086692 1461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:27.097144 1467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:27.097157 1469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:27.097148 1466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:27.097563 1461 server_base.cc:1034] running on GCE node
I20260924 20:13:27.098109 1461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:27.098863 1461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:27.100013 1461 hybrid_clock.cc:648] HybridClock initialized: now 1790280807099994 us; error 33 us; skew 500 ppm
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:27.102802 1461 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:27.103773 1461 webserver.cc:460] Webserver started at http://127.29.83.131:46059/ using document root <none> and password file <none>
I20260924 20:13:27.104382 1461 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:27.104444 1461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:27.104606 1461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:27.106792 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "ba459505ed5649b787d05e1949aa32a8"
format_stamp: "Formatted at 2026-09-24 20:13:27 on dist-test-slave-52fr"
server_key: "45eb9491045a27c0cffe46a8a9d969da"
server_key_iv: "357b3c27a74d28e75c9a26ffa44d3fcb"
server_key_version: "encryptionkey@0"
I20260924 20:13:27.107304 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "ba459505ed5649b787d05e1949aa32a8"
format_stamp: "Formatted at 2026-09-24 20:13:27 on dist-test-slave-52fr"
server_key: "45eb9491045a27c0cffe46a8a9d969da"
server_key_iv: "357b3c27a74d28e75c9a26ffa44d3fcb"
server_key_version: "encryptionkey@0"
I20260924 20:13:27.111270 1461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:27.113673 1476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:27.114955 1461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:27.115074 1461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "ba459505ed5649b787d05e1949aa32a8"
format_stamp: "Formatted at 2026-09-24 20:13:27 on dist-test-slave-52fr"
server_key: "45eb9491045a27c0cffe46a8a9d969da"
server_key_iv: "357b3c27a74d28e75c9a26ffa44d3fcb"
server_key_version: "encryptionkey@0"
I20260924 20:13:27.115188 1461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:27.131224 1461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:27.134151 1461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:27.134343 1461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:27.134927 1461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:27.135711 1461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:27.135766 1461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:27.135807 1461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:27.135861 1461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:27.145244 1461 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:44145
I20260924 20:13:27.145267 1589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:44145 every 8 connection(s)
I20260924 20:13:27.146226 1461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:27.150919 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1461
I20260924 20:13:27.151171 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:27.151480 30030 external_mini_cluster.cc:1428] Setting key 6fc1bebb2e700deae5d46c8283f343f0
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:27.160157 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:27.148138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:60389 (local address 127.29.83.190:46229)
0924 20:13:27.148279 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:27.148282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:27.149126 (+ 844us) server_negotiation.cc:407] Connection header received
0924 20:13:27.150713 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:27.150716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:27.150782 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:27.150887 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:27.152306 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.152940 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.154093 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.154317 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.154428 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:27.156868 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:27.156872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:27.156874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:27.156897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:27.158229 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.158720 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.158725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.158727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.158806 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.159131 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.159134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.159136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.159307 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:27.159413 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:27.159877 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 20:13:27.159989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:27.160949 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:27.148475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46229 (local address 127.29.83.131:60389)
0924 20:13:27.148938 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:27.148981 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:27.150457 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:27.151068 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:27.151076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:27.151524 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:27.152151 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:27.152177 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.153067 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.153070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:27.153967 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:27.153976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.154464 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.154467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:27.154646 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:27.155263 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:27.155279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:27.156704 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:27.158347 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:27.158353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:27.158367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:27.158618 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:27.158896 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:27.158901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:27.158903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:27.159031 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:27.159408 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:27.159414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:27.159657 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 20:13:27.159841 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 20:13:27.160146 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":106,"threads_started":1}
I20260924 20:13:27.162046 1590 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46229
I20260924 20:13:27.162254 1590 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:27.162691 1590 heartbeater.cc:511] Master 127.29.83.190:46229 requested a full tablet report, sending...
I20260924 20:13:27.163640 1131 ts_manager.cc:194] Registered new tserver with Master: ba459505ed5649b787d05e1949aa32a8 (127.29.83.131:44145)
I20260924 20:13:27.164359 1131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:60389
I20260924 20:13:27.165283 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:27.190737 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:27.181192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.29.83.129:41195)
0924 20:13:27.181526 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:27.181540 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:27.181563 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:13:27.181614 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:27.181619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:27.181915 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:27.182074 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:27.182807 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.183692 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.184545 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.184947 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.185031 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:27.187332 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:27.187341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:27.187353 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:27.187376 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:27.189016 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.189417 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.189422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.189424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.189507 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.189791 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.189797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.189803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.190039 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:27.190222 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:27.190367 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 20:13:27.190517 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":147,"threads_started":1}
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:27.201367 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:27.193628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.29.83.190:46229)
0924 20:13:27.193808 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:27.193812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:27.193833 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:13:27.193920 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:27.193922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:27.193960 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:27.194023 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:27.194642 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.195145 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.195924 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:27.196127 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:27.196202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:27.198394 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:27.198396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:27.198398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:27.198421 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:27.199797 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.200246 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.200249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.200250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.200319 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:27.200532 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:27.200534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:27.200535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:27.200745 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:27.200845 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:27.201072 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 20:13:27.201183 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 20:13:27.201828 1131 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59300
I20260924 20:13:27.202505 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1193
I20260924 20:13:27.208316 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1327
I20260924 20:13:27.214254 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1461
I20260924 20:13:27.219485 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1099
2026-09-24T20:13:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2136 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1602](info): set up 2 sockets
Sep 24 20:13:27 dist-test-slave-52fr krb5kdc[1602](info): commencing operation
krb5kdc: starting...
W20260924 20:13:28.253991 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:28 dist-test-slave-52fr krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:28.375201 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34071
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:28.483177 1615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:28.483489 1615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:28.483532 1615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:28.487773 1615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:28.487857 1615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:28.487877 1615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:28.487895 1615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:28.487910 1615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:28.493624 1615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:34071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:28.494855 1615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:28.495936 1615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:28.505049 1623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:28.505082 1621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:28.505993 1615 server_base.cc:1034] running on GCE node
W20260924 20:13:28.506090 1620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:28.506698 1615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:28.507901 1615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:28.509099 1615 hybrid_clock.cc:648] HybridClock initialized: now 1790280808509080 us; error 31 us; skew 500 ppm
Sep 24 20:13:28 dist-test-slave-52fr krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280808, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:28.512101 1615 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:28.512868 1615 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:28.512965 1615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:28.513206 1615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:28.515475 1615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "f08cf3092b7349abbc94814506d3f5d4"
format_stamp: "Formatted at 2026-09-24 20:13:28 on dist-test-slave-52fr"
server_key: "562f97a3f059226871d6be05ac17876a"
server_key_iv: "5f8e33beb30985a8784bbe416ac67817"
server_key_version: "encryptionkey@0"
I20260924 20:13:28.516057 1615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "f08cf3092b7349abbc94814506d3f5d4"
format_stamp: "Formatted at 2026-09-24 20:13:28 on dist-test-slave-52fr"
server_key: "562f97a3f059226871d6be05ac17876a"
server_key_iv: "5f8e33beb30985a8784bbe416ac67817"
server_key_version: "encryptionkey@0"
I20260924 20:13:28.519917 1615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:28.522545 1629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:28.523887 1615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:28.524062 1615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "f08cf3092b7349abbc94814506d3f5d4"
format_stamp: "Formatted at 2026-09-24 20:13:28 on dist-test-slave-52fr"
server_key: "562f97a3f059226871d6be05ac17876a"
server_key_iv: "5f8e33beb30985a8784bbe416ac67817"
server_key_version: "encryptionkey@0"
I20260924 20:13:28.524205 1615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:28.541549 1615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:28.544797 1615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:28.545024 1615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:28.552177 1615 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:34071
I20260924 20:13:28.552475 1682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:34071 every 8 connection(s)
I20260924 20:13:28.553272 1615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:28.556576 1683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:28.561712 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1615
I20260924 20:13:28.561888 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:28.562209 30030 external_mini_cluster.cc:1428] Setting key 7c05bd89da7308425bfc942f863dad40
I20260924 20:13:28.564455 1683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: Bootstrap starting.
I20260924 20:13:28.566913 1683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:28.567708 1683 log.cc:824] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:28.569868 1683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: No bootstrap required, opened a new log
Sep 24 20:13:28 dist-test-slave-52fr krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:28.574084 1683 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } }
I20260924 20:13:28.574349 1683 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:28.574416 1683 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08cf3092b7349abbc94814506d3f5d4, State: Initialized, Role: FOLLOWER
I20260924 20:13:28.574927 1683 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } }
I20260924 20:13:28.575124 1683 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:28.575196 1683 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:28.575299 1683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:28.576542 1683 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } }
I20260924 20:13:28.576962 1683 leader_election.cc:302] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08cf3092b7349abbc94814506d3f5d4; no voters:
I20260924 20:13:28.577456 1683 leader_election.cc:288] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:28.577728 1688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:28.577975 1688 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [term 1 LEADER]: Becoming Leader. State: Replica: f08cf3092b7349abbc94814506d3f5d4, State: Running, Role: LEADER
I20260924 20:13:28.578358 1688 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } }
I20260924 20:13:28.578787 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:28.563797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.29.83.190:34071)
0924 20:13:28.564281 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:28.564302 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:28.564340 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:13:28.565083 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:28.565104 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:28.565456 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:28.565844 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:28.566683 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:28.567750 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:28.568672 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:28.569164 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:28.569352 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:28.572120 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:28.572125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:28.572140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:28.572168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:28.574903 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:28.575468 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:28.575477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:28.575486 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:28.575641 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:28.575972 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:28.575978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:28.575983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:28.576553 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:28.576781 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:28.577130 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 20:13:28.577626 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260924 20:13:28.578897 1683 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:28.579977 1688 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08cf3092b7349abbc94814506d3f5d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } } }
I20260924 20:13:28.580096 1688 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:28.580142 1689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08cf3092b7349abbc94814506d3f5d4. Latest consensus state: current_term: 1 leader_uuid: "f08cf3092b7349abbc94814506d3f5d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08cf3092b7349abbc94814506d3f5d4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34071 } } }
I20260924 20:13:28.580255 1689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:28.581408 1695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:28.584213 1695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:28.590240 1695 catalog_manager.cc:1347] Generated new cluster ID: f993961ecee2457fa7da9e8d5c8ba485
I20260924 20:13:28.590332 1695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:28.600183 1695 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:28.601473 1695 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:28.608500 1695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: Generated new TSK 0
I20260924 20:13:28.609385 1695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
W20260924 20:13:29.538913 30030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.83.129: real 0.919s user 0.000s sys 0.003s
I20260924 20:13:29.539458 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:29.654068 1709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:29.654356 1709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:29.654402 1709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:29.658602 1709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:29.658674 1709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:29.658737 1709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:29.664055 1709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:29.665135 1709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:29.666260 1709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:29.676319 1717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:29.676330 1714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:29.676319 1715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:29.676738 1709 server_base.cc:1034] running on GCE node
I20260924 20:13:29.677243 1709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:29.677942 1709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:29.679123 1709 hybrid_clock.cc:648] HybridClock initialized: now 1790280809679097 us; error 38 us; skew 500 ppm
Sep 24 20:13:29 dist-test-slave-52fr krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280809, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:29.682224 1709 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:29.683010 1709 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:29.683075 1709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:29.683290 1709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:29.685405 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "6b34f657aba24886be86d399eece3234"
format_stamp: "Formatted at 2026-09-24 20:13:29 on dist-test-slave-52fr"
server_key: "25e064dce727c545551eaa7344ffbfb8"
server_key_iv: "410cd8e6cb15e81cc9fec9f0b1decb89"
server_key_version: "encryptionkey@0"
I20260924 20:13:29.686010 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "6b34f657aba24886be86d399eece3234"
format_stamp: "Formatted at 2026-09-24 20:13:29 on dist-test-slave-52fr"
server_key: "25e064dce727c545551eaa7344ffbfb8"
server_key_iv: "410cd8e6cb15e81cc9fec9f0b1decb89"
server_key_version: "encryptionkey@0"
I20260924 20:13:29.689983 1709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:13:29.692474 1723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:29.693912 1709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:29.694044 1709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "6b34f657aba24886be86d399eece3234"
format_stamp: "Formatted at 2026-09-24 20:13:29 on dist-test-slave-52fr"
server_key: "25e064dce727c545551eaa7344ffbfb8"
server_key_iv: "410cd8e6cb15e81cc9fec9f0b1decb89"
server_key_version: "encryptionkey@0"
I20260924 20:13:29.694155 1709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:29.712643 1709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:29.716168 1709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:29.716387 1709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:29.717001 1709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:29.718029 1709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:29.718096 1709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:29.718173 1709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:29.718215 1709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:29.728636 1709 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:39541
I20260924 20:13:29.728655 1836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:39541 every 8 connection(s)
I20260924 20:13:29.729252 1709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:29.736797 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1709
I20260924 20:13:29.736935 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:29.737260 30030 external_mini_cluster.cc:1428] Setting key 0fca4ef6cd0def6f7f3480596ed59592
Sep 24 20:13:29 dist-test-slave-52fr krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280809, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:29.744643 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:29.731016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:58037 (local address 127.29.83.190:34071)
0924 20:13:29.731249 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:29.731253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:29.732019 (+ 766us) server_negotiation.cc:407] Connection header received
0924 20:13:29.733411 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:29.733414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:29.733479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:29.733591 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:29.734801 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:29.735393 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:29.736551 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:29.736802 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:29.736916 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:29.740023 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:29.740025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:29.740028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:29.740052 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:29.742247 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:29.742816 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:29.742822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:29.742828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:29.742928 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:29.743251 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:29.743258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:29.743262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:29.743533 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:29.743645 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:29.744237 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 20:13:29.744388 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260924 20:13:29.745368 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:29.731347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:34071 (local address 127.29.83.129:58037)
0924 20:13:29.731807 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:29.731919 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:29.733163 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:29.733821 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:29.733828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:29.734187 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:29.734650 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:29.734663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:29.735512 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:29.735516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:29.736414 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:29.736423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:29.736953 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:29.736957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:29.737157 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:29.737823 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:29.737845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:29.739829 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:29.742380 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:29.742385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:29.742399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:29.742711 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:29.743022 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:29.743024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:29.743026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:29.743161 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:29.743666 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:29.743700 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:29.743984 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 20:13:29.744159 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 20:13:29.744427 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260924 20:13:29.746814 1837 heartbeater.cc:347] Connected to a master server at 127.29.83.190:34071
I20260924 20:13:29.747061 1837 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:29.747529 1837 heartbeater.cc:511] Master 127.29.83.190:34071 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:29.749075 1646 ts_manager.cc:194] Registered new tserver with Master: 6b34f657aba24886be86d399eece3234 (127.29.83.129:39541)
I20260924 20:13:29.750216 1646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:58037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
W20260924 20:13:30.487149 30030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.83.130: real 0.748s user 0.001s sys 0.002s
I20260924 20:13:30.487709 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:30.603044 1843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:30.603341 1843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:30.603464 1843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:30.607662 1843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:30.607774 1843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:30.607875 1843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:30.613374 1843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:30.614557 1843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:30.615624 1843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:30.625483 1851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:30.625499 1849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:30.625568 1843 server_base.cc:1034] running on GCE node
W20260924 20:13:30.625504 1848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:30.626178 1843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:30.626874 1843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:30.628108 1843 hybrid_clock.cc:648] HybridClock initialized: now 1790280810628071 us; error 62 us; skew 500 ppm
Sep 24 20:13:30 dist-test-slave-52fr krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:30.631115 1843 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:30.631892 1843 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:30.631990 1843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:30.632229 1843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:30.634457 1843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "4dd5a42f00ea4fa0981ec4365345d61c"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "60000e81369e00e93ceda066e1f19d4a"
server_key_iv: "f5feac19dfd6e7c5f17f292a3df6b39d"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.635035 1843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "4dd5a42f00ea4fa0981ec4365345d61c"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "60000e81369e00e93ceda066e1f19d4a"
server_key_iv: "f5feac19dfd6e7c5f17f292a3df6b39d"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.638862 1843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:13:30.641364 1857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.642772 1843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:30.642927 1843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "4dd5a42f00ea4fa0981ec4365345d61c"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "60000e81369e00e93ceda066e1f19d4a"
server_key_iv: "f5feac19dfd6e7c5f17f292a3df6b39d"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.643050 1843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:30.655404 1843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:30.658601 1843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:30.658809 1843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:30.659408 1843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:30.660248 1843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:30.660326 1843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.660400 1843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:30.660437 1843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.669445 1843 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:35431
I20260924 20:13:30.669464 1970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:35431 every 8 connection(s)
I20260924 20:13:30.670051 1843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:30.674772 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1843
I20260924 20:13:30.674872 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:30.675196 30030 external_mini_cluster.cc:1428] Setting key 4a2a24ab1cb42ac316c78a4ccbdbb760
Sep 24 20:13:30 dist-test-slave-52fr krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:30.685539 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.671802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:38579 (local address 127.29.83.190:34071)
0924 20:13:30.672034 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.672038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.672897 (+ 859us) server_negotiation.cc:407] Connection header received
0924 20:13:30.674268 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.674272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.674321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.674412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:30.676019 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.676763 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.677945 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.678349 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.678475 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.681186 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:30.681192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:30.681198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:30.681232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:30.683174 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.683765 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.683769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.683771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.683876 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.684253 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.684259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.684264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.684533 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:30.684654 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.685207 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.685349 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:30.686411 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.672235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:34071 (local address 127.29.83.130:38579)
0924 20:13:30.672735 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:30.672780 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:30.674051 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:30.674599 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:30.674606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:30.675136 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:30.675807 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.675833 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.676891 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.676895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.677831 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.677840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.678501 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.678504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.678658 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.679273 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:30.679300 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:30.681014 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:30.683347 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:30.683352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:30.683365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:30.683642 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:30.683995 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:30.683998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:30.683999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:30.684147 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:30.684668 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:30.684674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:30.684927 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 20:13:30.685111 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 20:13:30.685404 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260924 20:13:30.687700 1971 heartbeater.cc:347] Connected to a master server at 127.29.83.190:34071
I20260924 20:13:30.688019 1971 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:13:30.688583 1971 heartbeater.cc:511] Master 127.29.83.190:34071 requested a full tablet report, sending...
I20260924 20:13:30.689733 1646 ts_manager.cc:194] Registered new tserver with Master: 4dd5a42f00ea4fa0981ec4365345d61c (127.29.83.130:35431)
I20260924 20:13:30.690363 1646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:38579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:30.703024 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:13:30.752348 1837 heartbeater.cc:503] Master 127.29.83.190:34071 was elected leader, sending a full tablet report...
W20260924 20:13:30.814393 1977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:30.814715 1977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:30.814795 1977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:30.818974 1977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:30.819064 1977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:30.819167 1977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:30.824904 1977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:30.826097 1977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:30.827173 1977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:30.837209 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:30.837241 1985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:30.837325 1977 server_base.cc:1034] running on GCE node
W20260924 20:13:30.837241 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:30.837970 1977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:30.838632 1977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:30.839797 1977 hybrid_clock.cc:648] HybridClock initialized: now 1790280810839783 us; error 26 us; skew 500 ppm
Sep 24 20:13:30 dist-test-slave-52fr krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:30.842749 1977 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:30.843448 1977 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:30.843513 1977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:30.843677 1977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:30.845693 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "05f4c5af48de4301b61a9634d5f5114b"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "71c534f80614cd826c2a86bc610c0d57"
server_key_iv: "f9e7097970df6c89679ac3b2afbf94cf"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.846540 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "05f4c5af48de4301b61a9634d5f5114b"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "71c534f80614cd826c2a86bc610c0d57"
server_key_iv: "f9e7097970df6c89679ac3b2afbf94cf"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.850804 1977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:30.853329 1991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.854806 1977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:30.854952 1977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "05f4c5af48de4301b61a9634d5f5114b"
format_stamp: "Formatted at 2026-09-24 20:13:30 on dist-test-slave-52fr"
server_key: "71c534f80614cd826c2a86bc610c0d57"
server_key_iv: "f9e7097970df6c89679ac3b2afbf94cf"
server_key_version: "encryptionkey@0"
I20260924 20:13:30.855082 1977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:30.871449 1977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:30.874887 1977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:30.875090 1977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:30.875696 1977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:30.876535 1977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:30.876588 1977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.876629 1977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:30.876684 1977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:30.886559 1977 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42345
I20260924 20:13:30.886574 2104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42345 every 8 connection(s)
I20260924 20:13:30.887168 1977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:30.889936 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 1977
I20260924 20:13:30.890067 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:30.890398 30030 external_mini_cluster.cc:1428] Setting key 5bef1ed22c3ee7a84600ac964b26277d
Sep 24 20:13:30 dist-test-slave-52fr krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:30.901101 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.889018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:42565 (local address 127.29.83.190:34071)
0924 20:13:30.889165 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.889168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.890083 (+ 915us) server_negotiation.cc:407] Connection header received
0924 20:13:30.891730 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.891733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.891798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.891883 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:30.893123 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.893651 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.894834 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.895063 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.895155 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.897602 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:30.897605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:30.897607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:30.897633 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:30.899134 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.899608 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.899611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.899612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.899684 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.900050 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.900052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.900054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.900245 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:30.900331 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.900859 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.900974 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:30.902117 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.889413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:34071 (local address 127.29.83.131:42565)
0924 20:13:30.889925 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:30.889969 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:30.891504 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:30.892051 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:30.892058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:30.892517 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:30.893001 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.893013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.893810 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.893813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.894705 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.894713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.895213 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.895216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.895369 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.895945 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:30.895962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:30.897405 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:30.899249 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:30.899254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:30.899268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:30.899522 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:30.899775 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:30.899777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:30.899778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:30.899908 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:30.900373 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:30.900379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:30.900628 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 20:13:30.900808 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 20:13:30.901077 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":152,"threads_started":1}
I20260924 20:13:30.903538 2105 heartbeater.cc:347] Connected to a master server at 127.29.83.190:34071
I20260924 20:13:30.903829 2105 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:30.904328 2105 heartbeater.cc:511] Master 127.29.83.190:34071 requested a full tablet report, sending...
I20260924 20:13:30.905256 1646 ts_manager.cc:194] Registered new tserver with Master: 05f4c5af48de4301b61a9634d5f5114b (127.29.83.131:42345)
I20260924 20:13:30.905865 1646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:42565
I20260924 20:13:30.916172 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:13:30.926653 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.918116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.29.83.190:34071)
0924 20:13:30.918310 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.918316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.918482 (+ 166us) server_negotiation.cc:407] Connection header received
0924 20:13:30.918662 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.918665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.918735 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.918821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:30.919543 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.920200 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.921319 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.921564 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.921686 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.923367 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:30.923371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:30.923374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:30.923401 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:30.924720 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.925307 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.925311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.925313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.925404 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:30.925729 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:30.925732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:30.925734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:30.925991 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:30.926159 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.926375 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.926492 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:13:30.929697 1646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:30.932701 1646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:30.947764 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.941631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35431 (local address 127.0.0.1:49418)
0924 20:13:30.942774 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:30.942831 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:30.942969 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:30.943572 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:30.943578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:30.943600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:30.943742 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.943751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.944869 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.944873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.946125 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.946135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947186 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.947310 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.947351 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 20:13:30.947463 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 20:13:30.947584 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":146,"threads_started":1}
I20260924 20:13:30.948362 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.942635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.29.83.131:42345)
0924 20:13:30.943197 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.943210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.943232 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:13:30.943305 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.943308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.943452 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.943612 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:30.944030 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.945260 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.946847 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947749 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.948029 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.948108 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.948204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":160,"threads_started":1}
I20260924 20:13:30.948566 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.942025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.29.83.129:39541)
0924 20:13:30.942350 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.942359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.942962 (+ 603us) server_negotiation.cc:407] Connection header received
0924 20:13:30.943360 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.943366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.943527 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.943626 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:30.944133 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.945255 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.946914 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947671 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947767 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.947924 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.948328 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.948428 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260924 20:13:30.948383 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.941845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.29.83.130:35431)
0924 20:13:30.942204 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:30.942215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:30.943108 (+ 893us) server_negotiation.cc:407] Connection header received
0924 20:13:30.943193 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:30.943199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:30.943412 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:30.943559 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:30.943872 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.944739 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.946283 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947061 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947173 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.947366 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:30.948127 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 20:13:30.948228 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260924 20:13:30.948262 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.942491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42345 (local address 127.0.0.1:40990)
0924 20:13:30.943048 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:30.943064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:30.943147 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:30.943764 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:30.943767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:30.943787 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:30.943921 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.943927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.945400 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.945403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.946715 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.946725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947879 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.947970 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.947992 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:13:30.948052 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:13:30.948131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":47,"thread_start_us":97,"threads_started":1}
I20260924 20:13:30.948427 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:30.942302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39541 (local address 127.0.0.1:41172)
0924 20:13:30.942872 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:30.942885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:30.943204 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:30.943793 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:30.943795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:30.943836 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:30.944009 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.944016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.945400 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.945403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.946826 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:30.946833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:30.947982 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:30.947986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:30.948085 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:30.948109 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:13:30.948177 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 20:13:30.948254 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":79,"thread_start_us":82,"threads_started":1}
I20260924 20:13:30.951647 2039 tablet_service.cc:1467] Processing CreateTablet for tablet da8550de68b1402cb6062d99ccb76433 (DEFAULT_TABLE table=test-table [id=09f2ac3eec644c1dae21c8ce83366a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:30.951799 1770 tablet_service.cc:1467] Processing CreateTablet for tablet da8550de68b1402cb6062d99ccb76433 (DEFAULT_TABLE table=test-table [id=09f2ac3eec644c1dae21c8ce83366a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:30.951799 1905 tablet_service.cc:1467] Processing CreateTablet for tablet da8550de68b1402cb6062d99ccb76433 (DEFAULT_TABLE table=test-table [id=09f2ac3eec644c1dae21c8ce83366a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:30.952666 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8550de68b1402cb6062d99ccb76433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:30.952795 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8550de68b1402cb6062d99ccb76433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:30.952795 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8550de68b1402cb6062d99ccb76433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:30.958762 2123 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Bootstrap starting.
I20260924 20:13:30.960242 2125 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Bootstrap starting.
I20260924 20:13:30.960909 2124 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Bootstrap starting.
I20260924 20:13:30.961350 2123 tablet_bootstrap.cc:654] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:30.962725 2125 tablet_bootstrap.cc:654] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:30.962749 2124 tablet_bootstrap.cc:654] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:30.962764 2123 log.cc:824] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:30.963783 2124 log.cc:824] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:30.963963 2125 log.cc:824] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:30.964921 2123 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: No bootstrap required, opened a new log
I20260924 20:13:30.965109 2123 ts_tablet_manager.cc:1397] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 20:13:30.965670 2125 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: No bootstrap required, opened a new log
I20260924 20:13:30.965857 2124 tablet_bootstrap.cc:492] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: No bootstrap required, opened a new log
I20260924 20:13:30.965885 2125 ts_tablet_manager.cc:1397] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:13:30.966029 2124 ts_tablet_manager.cc:1397] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260924 20:13:30.968729 2125 raft_consensus.cc:348] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.969026 2125 raft_consensus.cc:374] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:30.969081 2125 raft_consensus.cc:729] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd5a42f00ea4fa0981ec4365345d61c, State: Initialized, Role: FOLLOWER
I20260924 20:13:30.969158 2123 raft_consensus.cc:348] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.969419 2123 raft_consensus.cc:374] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:30.969472 2123 raft_consensus.cc:729] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4c5af48de4301b61a9634d5f5114b, State: Initialized, Role: FOLLOWER
I20260924 20:13:30.969592 2125 consensus_queue.cc:260] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.969952 2123 consensus_queue.cc:260] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.970376 1971 heartbeater.cc:503] Master 127.29.83.190:34071 was elected leader, sending a full tablet report...
I20260924 20:13:30.970644 2125 ts_tablet_manager.cc:1428] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:13:30.970877 2123 ts_tablet_manager.cc:1428] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260924 20:13:30.971174 2105 heartbeater.cc:503] Master 127.29.83.190:34071 was elected leader, sending a full tablet report...
I20260924 20:13:30.971509 2124 raft_consensus.cc:348] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.971781 2124 raft_consensus.cc:374] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:30.971835 2124 raft_consensus.cc:729] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b34f657aba24886be86d399eece3234, State: Initialized, Role: FOLLOWER
I20260924 20:13:30.972214 2124 consensus_queue.cc:260] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:30.973335 2124 ts_tablet_manager.cc:1428] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
W20260924 20:13:30.980939 1838 tablet.cc:2369] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:31.138090 2106 tablet.cc:2369] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:31.171178 1972 tablet.cc:2369] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:31.246475 2129 raft_consensus.cc:482] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:31.246707 2129 raft_consensus.cc:504] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:31.248051 2129 leader_election.cc:288] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b34f657aba24886be86d399eece3234 (127.29.83.129:39541), 05f4c5af48de4301b61a9634d5f5114b (127.29.83.131:42345)
I20260924 20:13:31.252031 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.248467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:39717 (local address 127.29.83.131:42345)
0924 20:13:31.248616 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.248621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.248796 (+ 175us) server_negotiation.cc:407] Connection header received
0924 20:13:31.248908 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.248910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.248976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.249054 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:31.249351 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.249932 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251052 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.251467 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251598 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:31.251654 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.251829 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.251903 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:13:31.252048 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.248246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39541 (local address 127.29.83.130:40231)
0924 20:13:31.248564 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:31.248577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:31.248702 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:31.248975 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:31.248978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:31.248995 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:31.249138 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:31.249145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.250000 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.250006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:31.250925 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:31.250934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.251619 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:31.251740 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:31.251757 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:13:31.251854 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 20:13:31.251919 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260924 20:13:31.252067 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.248375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:40231 (local address 127.29.83.129:39541)
0924 20:13:31.248500 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.248504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.248675 (+ 171us) server_negotiation.cc:407] Connection header received
0924 20:13:31.248811 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.248815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.248871 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.248947 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:31.249283 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.249894 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251084 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.251478 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251609 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:31.251659 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.251888 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.251963 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:13:31.252055 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.248318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42345 (local address 127.29.83.130:39717)
0924 20:13:31.248689 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:31.248704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:31.248813 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:31.249087 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:31.249089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:31.249103 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:31.249232 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:31.249238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.250064 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.250067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:31.250920 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:31.250929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.251596 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.251600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:31.251695 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:31.251717 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:13:31.251787 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:13:31.251875 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":27,"thread_start_us":82,"threads_started":1}
I20260924 20:13:31.252714 1788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8550de68b1402cb6062d99ccb76433" candidate_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b34f657aba24886be86d399eece3234" is_pre_election: true
I20260924 20:13:31.253089 1788 raft_consensus.cc:2454] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd5a42f00ea4fa0981ec4365345d61c in term 0.
I20260924 20:13:31.253296 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8550de68b1402cb6062d99ccb76433" candidate_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4c5af48de4301b61a9634d5f5114b" is_pre_election: true
I20260924 20:13:31.253571 2059 raft_consensus.cc:2454] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd5a42f00ea4fa0981ec4365345d61c in term 0.
I20260924 20:13:31.253587 1861 leader_election.cc:302] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd5a42f00ea4fa0981ec4365345d61c, 6b34f657aba24886be86d399eece3234; no voters:
I20260924 20:13:31.253881 2129 raft_consensus.cc:2781] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:31.253988 2129 raft_consensus.cc:482] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:31.254050 2129 raft_consensus.cc:3037] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:31.255059 2129 raft_consensus.cc:504] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:31.255404 2129 leader_election.cc:288] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [CANDIDATE]: Term 1 election: Requested vote from peers 6b34f657aba24886be86d399eece3234 (127.29.83.129:39541), 05f4c5af48de4301b61a9634d5f5114b (127.29.83.131:42345)
I20260924 20:13:31.255749 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8550de68b1402cb6062d99ccb76433" candidate_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4c5af48de4301b61a9634d5f5114b"
I20260924 20:13:31.255766 1788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8550de68b1402cb6062d99ccb76433" candidate_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b34f657aba24886be86d399eece3234"
I20260924 20:13:31.255872 1788 raft_consensus.cc:3037] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:31.255872 2059 raft_consensus.cc:3037] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:31.257165 1788 raft_consensus.cc:2454] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd5a42f00ea4fa0981ec4365345d61c in term 1.
I20260924 20:13:31.257165 2059 raft_consensus.cc:2454] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd5a42f00ea4fa0981ec4365345d61c in term 1.
I20260924 20:13:31.257471 1859 leader_election.cc:302] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4c5af48de4301b61a9634d5f5114b, 4dd5a42f00ea4fa0981ec4365345d61c; no voters:
I20260924 20:13:31.257656 2129 raft_consensus.cc:2781] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:31.257921 2129 raft_consensus.cc:686] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 1 LEADER]: Becoming Leader. State: Replica: 4dd5a42f00ea4fa0981ec4365345d61c, State: Running, Role: LEADER
I20260924 20:13:31.258260 2129 consensus_queue.cc:237] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } }
I20260924 20:13:31.261605 1646 catalog_manager.cc:5495] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd5a42f00ea4fa0981ec4365345d61c (127.29.83.130). New cstate: current_term: 1 leader_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd5a42f00ea4fa0981ec4365345d61c" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:31.316704 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.313258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.29.83.130:35431)
0924 20:13:31.313386 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.313390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.313403 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:13:31.313475 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.313477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.313528 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.313614 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:31.313940 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.314447 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.315239 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:31.315444 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:31.315521 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:31.315790 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:31.316206 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:31.316368 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.316460 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.316510 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:13:31.322638 1788 raft_consensus.cc:1261] T da8550de68b1402cb6062d99ccb76433 P 6b34f657aba24886be86d399eece3234 [term 1 FOLLOWER]: Refusing update from remote peer 4dd5a42f00ea4fa0981ec4365345d61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:31.322638 2059 raft_consensus.cc:1261] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 1 FOLLOWER]: Refusing update from remote peer 4dd5a42f00ea4fa0981ec4365345d61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:31.323374 2134 consensus_queue.cc:1035] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b34f657aba24886be86d399eece3234" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:31.323549 2129 consensus_queue.cc:1035] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4c5af48de4301b61a9634d5f5114b" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:31.330737 2137 mvcc.cc:205] Tried to move back new op lower bound from 7332990203170058240 to 7332990202916708352. Current Snapshot: MvccSnapshot[applied={T|T < 7332990203170058240}]
I20260924 20:13:31.332653 2138 mvcc.cc:205] Tried to move back new op lower bound from 7332990203170058240 to 7332990202916708352. Current Snapshot: MvccSnapshot[applied={T|T < 7332990203170058240}]
I20260924 20:13:31.332917 2139 mvcc.cc:205] Tried to move back new op lower bound from 7332990203170058240 to 7332990202916708352. Current Snapshot: MvccSnapshot[applied={T|T < 7332990203170058240 or (T in {7332990203170058240})}]
I20260924 20:13:31.341625 1645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:31.341851 1645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:31.344406 1645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f08cf3092b7349abbc94814506d3f5d4: Sending DeleteTablet for 3 replicas of tablet da8550de68b1402cb6062d99ccb76433
I20260924 20:13:31.345276 2039 tablet_service.cc:1514] Processing DeleteTablet for tablet da8550de68b1402cb6062d99ccb76433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:31 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:40990
I20260924 20:13:31.345293 1770 tablet_service.cc:1514] Processing DeleteTablet for tablet da8550de68b1402cb6062d99ccb76433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:31 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:41172
I20260924 20:13:31.345857 2147 tablet_replica.cc:330] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: stopping tablet replica
I20260924 20:13:31.346006 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1709
I20260924 20:13:31.345958 1905 tablet_service.cc:1514] Processing DeleteTablet for tablet da8550de68b1402cb6062d99ccb76433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:31 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:49418
I20260924 20:13:31.346248 2147 raft_consensus.cc:2229] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:31.347077 2147 raft_consensus.cc:2258] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:31.347720 2149 tablet_replica.cc:330] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: stopping tablet replica
I20260924 20:13:31.348014 2149 raft_consensus.cc:2229] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:31.348423 2149 raft_consensus.cc:2258] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:31.348762 2147 ts_tablet_manager.cc:1905] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:31.349643 2149 ts_tablet_manager.cc:1905] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:31.351711 2149 ts_tablet_manager.cc:1918] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:31.351831 2149 log.cc:1196] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/da8550de68b1402cb6062d99ccb76433
I20260924 20:13:31.352101 2149 ts_tablet_manager.cc:1939] T da8550de68b1402cb6062d99ccb76433 P 4dd5a42f00ea4fa0981ec4365345d61c: Deleting consensus metadata
I20260924 20:13:31.353276 1631 catalog_manager.cc:4841] TS 4dd5a42f00ea4fa0981ec4365345d61c (127.29.83.130:35431): tablet da8550de68b1402cb6062d99ccb76433 (table test-table [id=09f2ac3eec644c1dae21c8ce83366a39]) successfully deleted
I20260924 20:13:31.353935 2147 ts_tablet_manager.cc:1918] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:31.354049 2147 log.cc:1196] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/da8550de68b1402cb6062d99ccb76433
I20260924 20:13:31.354329 2147 ts_tablet_manager.cc:1939] T da8550de68b1402cb6062d99ccb76433 P 05f4c5af48de4301b61a9634d5f5114b: Deleting consensus metadata
I20260924 20:13:31.355422 1631 catalog_manager.cc:4841] TS 05f4c5af48de4301b61a9634d5f5114b (127.29.83.131:42345): tablet da8550de68b1402cb6062d99ccb76433 (table test-table [id=09f2ac3eec644c1dae21c8ce83366a39]) successfully deleted
W20260924 20:13:31.357228 1633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:39541 (error 108)
I20260924 20:13:31.357393 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1843
I20260924 20:13:31.362587 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.362314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39541 (local address 127.0.0.1:41178)
0924 20:13:31.362449 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:31.362511 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 20:13:31.362756 1633 catalog_manager.cc:4584] TS 6b34f657aba24886be86d399eece3234 (127.29.83.129:39541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da8550de68b1402cb6062d99ccb76433: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39541: connect: Connection refused (error 111)
I20260924 20:13:31.364012 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1977
I20260924 20:13:31.369665 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 1615
2026-09-24T20:13:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4149 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T20:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:31Z Disabled control of system clock
I20260924 20:13:31.399384 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38627
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:13:31.509482 2154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:31.509814 2154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:31.509871 2154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:31.513824 2154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:31.513880 2154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:31.513895 2154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:31.513950 2154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:31.513983 2154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:31.520195 2154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:31.521302 2154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:31.522832 2154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:31.531661 2160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:31.531694 2154 server_base.cc:1034] running on GCE node
W20260924 20:13:31.531641 2162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:31.531631 2159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:31.532456 2154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:31.533607 2154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:31.534787 2154 hybrid_clock.cc:648] HybridClock initialized: now 1790280811534769 us; error 36 us; skew 500 ppm
I20260924 20:13:31.543079 2154 webserver.cc:460] Webserver started at http://127.29.83.190:35941/ using document root <none> and password file <none>
I20260924 20:13:31.543745 2154 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:31.543812 2154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:31.544059 2154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:31.546262 2154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "54dae85f8bf347a7898b3ac8de7f4e8b"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "a82075e88c69d139da3d2717252b3ad5"
server_key_iv: "f191d31c7ff6b4065d8c9f7c0341cbbc"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.546847 2154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "54dae85f8bf347a7898b3ac8de7f4e8b"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "a82075e88c69d139da3d2717252b3ad5"
server_key_iv: "f191d31c7ff6b4065d8c9f7c0341cbbc"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.551174 2154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:31.553727 2168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.555135 2154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:31.555297 2154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "54dae85f8bf347a7898b3ac8de7f4e8b"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "a82075e88c69d139da3d2717252b3ad5"
server_key_iv: "f191d31c7ff6b4065d8c9f7c0341cbbc"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.555428 2154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:31.560132 2154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:31.560773 2154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:31.560959 2154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:31.568483 2154 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38627
I20260924 20:13:31.568501 2220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38627 every 8 connection(s)
I20260924 20:13:31.569506 2154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:31.572849 2221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:31.575148 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2154
I20260924 20:13:31.575263 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:31.575552 30030 external_mini_cluster.cc:1428] Setting key 820a5fc2a643fb13f0170d3d0f0110ff
I20260924 20:13:31.579738 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: Bootstrap starting.
I20260924 20:13:31.581555 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.577238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.29.83.190:38627)
0924 20:13:31.578000 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.578016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.578052 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:13:31.578760 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.578778 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.579153 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.579518 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:31.580290 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:31.580295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:31.580318 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:31.580327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:31.580381 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:31.580397 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:31.580555 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.580626 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.580877 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260924 20:13:31.582258 2221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:31.583160 2221 log.cc:824] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:31.585239 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: No bootstrap required, opened a new log
I20260924 20:13:31.589126 2221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } }
I20260924 20:13:31.589447 2221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:31.589541 2221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dae85f8bf347a7898b3ac8de7f4e8b, State: Initialized, Role: FOLLOWER
I20260924 20:13:31.590072 2221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } }
I20260924 20:13:31.590231 2221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:31.590301 2221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:31.590420 2221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:31.591427 2221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } }
I20260924 20:13:31.591777 2221 leader_election.cc:302] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54dae85f8bf347a7898b3ac8de7f4e8b; no voters:
I20260924 20:13:31.592083 2221 leader_election.cc:288] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:31.592185 2226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:31.592444 2226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [term 1 LEADER]: Becoming Leader. State: Replica: 54dae85f8bf347a7898b3ac8de7f4e8b, State: Running, Role: LEADER
I20260924 20:13:31.592757 2226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } }
I20260924 20:13:31.593297 2221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:31.594466 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54dae85f8bf347a7898b3ac8de7f4e8b. Latest consensus state: current_term: 1 leader_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } } }
I20260924 20:13:31.594508 2227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dae85f8bf347a7898b3ac8de7f4e8b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38627 } } }
I20260924 20:13:31.594590 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:31.594609 2227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:31.595034 2235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:31.597702 2235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:31.603250 2235 catalog_manager.cc:1347] Generated new cluster ID: bf5084a8bf944bc18641a7b6695f0203
I20260924 20:13:31.603319 2235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:31.618180 2235 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:31.618589 2235 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:31.627110 2235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: Generated new TSK 0
I20260924 20:13:31.627964 2235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:13:31.639907 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--builtin_ntp_servers=127.29.83.148:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:13:31.753204 2245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:31.753495 2245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:31.753538 2245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:31.757716 2245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:31.757800 2245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:31.757908 2245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:31.762902 2245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:31.764017 2245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:31.765472 2245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:31.775933 2245 server_base.cc:1034] running on GCE node
W20260924 20:13:31.775895 2250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:31.775969 2251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:31.775895 2253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:31.776597 2245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:31.777215 2245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:31.778402 2245 hybrid_clock.cc:648] HybridClock initialized: now 1790280811778388 us; error 33 us; skew 500 ppm
I20260924 20:13:31.780452 2245 webserver.cc:460] Webserver started at http://127.29.83.129:40471/ using document root <none> and password file <none>
I20260924 20:13:31.781097 2245 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:31.781180 2245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:31.781409 2245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:31.783591 2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "9775c0921e1066c31ac8bedb340d1fca"
server_key_iv: "c00cb5a6a8ab573ef19578885ccea566"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.784195 2245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "9775c0921e1066c31ac8bedb340d1fca"
server_key_iv: "c00cb5a6a8ab573ef19578885ccea566"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.788331 2245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:13:31.791224 2259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.792671 2245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:13:31.792831 2245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "9775c0921e1066c31ac8bedb340d1fca"
server_key_iv: "c00cb5a6a8ab573ef19578885ccea566"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.792968 2245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:31.798036 2245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:31.798728 2245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:31.798921 2245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:31.799501 2245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:31.800447 2245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:31.800521 2245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.800603 2245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:31.800638 2245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.809995 2245 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:34125
I20260924 20:13:31.810014 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:34125 every 8 connection(s)
I20260924 20:13:31.810945 2245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:31.816849 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2245
I20260924 20:13:31.816991 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:31.817267 30030 external_mini_cluster.cc:1428] Setting key bd5feab8343a4ce930e294f11e2735e0
I20260924 20:13:31.818135 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.813152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:50701 (local address 127.29.83.190:38627)
0924 20:13:31.813323 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.813326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.814429 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 20:13:31.815969 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.815973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.816015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.816101 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:31.817518 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:31.817522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:31.817531 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:31.817545 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:31.817566 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:31.817575 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:31.817692 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.817938 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.817989 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:13:31.819159 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.813483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38627 (local address 127.29.83.129:50701)
0924 20:13:31.814255 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:31.814300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:31.815733 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:31.816295 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:31.816302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:31.816402 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:31.817124 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:31.817132 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:31.817184 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:31.817262 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:31.817706 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:31.817719 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:31.817726 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:13:31.817975 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 20:13:31.818264 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":112,"threads_started":1}
I20260924 20:13:31.819377 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--builtin_ntp_servers=127.29.83.148:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 20:13:31.820545 2373 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38627
I20260924 20:13:31.820847 2373 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:31.821295 2373 heartbeater.cc:511] Master 127.29.83.190:38627 requested a full tablet report, sending...
I20260924 20:13:31.822974 2185 ts_manager.cc:194] Registered new tserver with Master: a6a56ecf1b8d41a3b841e29b7a6996d8 (127.29.83.129:34125)
W20260924 20:13:31.931386 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:31.931655 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:31.931695 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:31.935564 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:31.935616 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:31.935671 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:31.940258 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:31.941269 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:31.942587 2376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:31.952088 2381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:31.952306 2376 server_base.cc:1034] running on GCE node
W20260924 20:13:31.952088 2382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:31.952134 2384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:31.952824 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:31.953490 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:31.954651 2376 hybrid_clock.cc:648] HybridClock initialized: now 1790280811954619 us; error 43 us; skew 500 ppm
I20260924 20:13:31.956641 2376 webserver.cc:460] Webserver started at http://127.29.83.130:32961/ using document root <none> and password file <none>
I20260924 20:13:31.957249 2376 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:31.957314 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:31.957566 2376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:31.959868 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "078cb3d4c1b048c58e444c7857d2e5e1"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "ec92f85cce4c308e3ca0d4b029f93036"
server_key_iv: "f39b34f0df33875fa20f01e28f54574e"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.960445 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "078cb3d4c1b048c58e444c7857d2e5e1"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "ec92f85cce4c308e3ca0d4b029f93036"
server_key_iv: "f39b34f0df33875fa20f01e28f54574e"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.964560 2376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:13:31.967351 2390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.968753 2376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:13:31.968905 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "078cb3d4c1b048c58e444c7857d2e5e1"
format_stamp: "Formatted at 2026-09-24 20:13:31 on dist-test-slave-52fr"
server_key: "ec92f85cce4c308e3ca0d4b029f93036"
server_key_iv: "f39b34f0df33875fa20f01e28f54574e"
server_key_version: "encryptionkey@0"
I20260924 20:13:31.969031 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:31.973845 2376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:31.974460 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:31.974653 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:31.975206 2376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:31.976150 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:31.976227 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.976291 2376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:31.976339 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:31.985533 2376 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:46753
I20260924 20:13:31.985551 2503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:46753 every 8 connection(s)
I20260924 20:13:31.986564 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:31.993000 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.988597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:59241 (local address 127.29.83.190:38627)
0924 20:13:31.988740 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:31.988746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:31.989816 (+ 1070us) server_negotiation.cc:407] Connection header received
0924 20:13:31.991170 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:31.991173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:31.991221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:31.991301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:31.992491 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:31.992494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:31.992497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:31.992511 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:31.992536 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:31.992552 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:31.992669 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:31.992810 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 20:13:31.992859 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:31.993985 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:31.988971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38627 (local address 127.29.83.130:59241)
0924 20:13:31.989642 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:31.989682 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:31.990949 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:31.991466 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:31.991473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:31.991565 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:31.992192 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:31.992194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:31.992245 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:31.992316 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:31.992649 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:31.992658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:31.992664 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:13:31.992857 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 20:13:31.993103 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260924 20:13:31.995381 2504 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38627
I20260924 20:13:31.995692 2504 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:31.996122 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2376
I20260924 20:13:31.996153 2504 heartbeater.cc:511] Master 127.29.83.190:38627 requested a full tablet report, sending...
I20260924 20:13:31.996282 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:31.996575 30030 external_mini_cluster.cc:1428] Setting key c6b8d276e4661aa4168afe9a03d31a1c
I20260924 20:13:31.997412 2185 ts_manager.cc:194] Registered new tserver with Master: 078cb3d4c1b048c58e444c7857d2e5e1 (127.29.83.130:46753)
I20260924 20:13:31.998654 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--builtin_ntp_servers=127.29.83.148:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:13:32.117286 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:32.117594 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:32.117637 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:32.121901 2507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:32.121973 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:32.122061 2507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:32.127427 2507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:32.128623 2507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:32.130169 2507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:32.140869 2507 server_base.cc:1034] running on GCE node
W20260924 20:13:32.140900 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:32.140887 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:32.141147 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:32.141592 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:32.142330 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:32.143518 2507 hybrid_clock.cc:648] HybridClock initialized: now 1790280812143498 us; error 33 us; skew 500 ppm
I20260924 20:13:32.145529 2507 webserver.cc:460] Webserver started at http://127.29.83.131:34425/ using document root <none> and password file <none>
I20260924 20:13:32.146204 2507 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:32.146268 2507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:32.146481 2507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:32.148542 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "7565c1d32565478cbda34c2c9e6d636c"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "cd27efec28d7ad9981968c932700174c"
server_key_iv: "87d27ddcca497ae5e3cc09dd61f0044e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.149118 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "7565c1d32565478cbda34c2c9e6d636c"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "cd27efec28d7ad9981968c932700174c"
server_key_iv: "87d27ddcca497ae5e3cc09dd61f0044e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.153041 2507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:13:32.156010 2521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:32.157403 2507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:32.157544 2507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "7565c1d32565478cbda34c2c9e6d636c"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "cd27efec28d7ad9981968c932700174c"
server_key_iv: "87d27ddcca497ae5e3cc09dd61f0044e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.157663 2507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:32.162448 2507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:32.163056 2507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:32.163251 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:32.163803 2507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:32.164773 2507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:32.164820 2507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:32.164860 2507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:32.164903 2507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:32.173907 2507 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:34555
I20260924 20:13:32.173924 2634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:34555 every 8 connection(s)
I20260924 20:13:32.174849 2507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:32.176854 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2507
I20260924 20:13:32.176985 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:32.177266 30030 external_mini_cluster.cc:1428] Setting key e70dc5c602fd87b3abbca6b90d2a3d66
I20260924 20:13:32.181438 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.176778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37187 (local address 127.29.83.190:38627)
0924 20:13:32.176959 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.176965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.178262 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 20:13:32.179860 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.179863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.179898 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.179962 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.181074 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.181077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.181079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.181089 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.181108 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.181115 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.181181 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.181302 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.181341 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:13:32.182415 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.177228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38627 (local address 127.29.83.131:37187)
0924 20:13:32.178066 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.178111 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.179671 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.180094 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.180100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.180195 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.180835 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.180836 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.180879 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.180950 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.181170 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.181197 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.181201 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.181323 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.181568 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":114,"threads_started":1}
I20260924 20:13:32.183524 2635 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38627
I20260924 20:13:32.183784 2635 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:32.184177 2635 heartbeater.cc:511] Master 127.29.83.190:38627 requested a full tablet report, sending...
I20260924 20:13:32.185145 2185 ts_manager.cc:194] Registered new tserver with Master: 7565c1d32565478cbda34c2c9e6d636c (127.29.83.131:34555)
I20260924 20:13:32.191710 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:13:32.195430 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.193626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.29.83.190:38627)
0924 20:13:32.193808 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.193814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.193924 (+ 110us) server_negotiation.cc:407] Connection header received
0924 20:13:32.194077 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.194079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.194113 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.194177 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.195022 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.195025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.195027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.195038 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.195057 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.195064 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.195140 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.195239 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.195271 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 20:13:32.198585 2185 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:32.201598 2185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:32.214164 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.211334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34125 (local address 127.0.0.1:42720)
0924 20:13:32.212122 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.212143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.212246 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.212621 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.212627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.212715 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.213311 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.213313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.213329 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.213382 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.213789 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.213801 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.213809 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.213916 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.213988 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":79,"threads_started":1}
I20260924 20:13:32.214314 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.211087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.29.83.130:46753)
0924 20:13:32.211375 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.211390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.211797 (+ 407us) server_negotiation.cc:407] Connection header received
0924 20:13:32.212005 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.212010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.212212 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.212324 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.213662 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.213668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.213687 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.213703 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.213749 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.213809 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.213909 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.214081 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.214164 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260924 20:13:32.214537 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.211109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.29.83.131:34555)
0924 20:13:32.211490 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.211503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.211792 (+ 289us) server_negotiation.cc:407] Connection header received
0924 20:13:32.212003 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.212010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.212212 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.212324 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.213921 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.213928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.213939 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.213951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.213991 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.214020 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.214097 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.214290 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.214364 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260924 20:13:32.214891 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.211724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.29.83.129:34125)
0924 20:13:32.212066 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.212078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.212229 (+ 151us) server_negotiation.cc:407] Connection header received
0924 20:13:32.212337 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.212343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.212484 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.212624 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.213509 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.213515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.213530 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.213545 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.213610 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.213646 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.214608 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.214662 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.214735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":146,"threads_started":1}
I20260924 20:13:32.214936 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.210602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:34555 (local address 127.0.0.1:36288)
0924 20:13:32.211650 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.211706 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.211880 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.213036 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.213040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.213057 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.213687 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.213689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.213696 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.213739 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.214166 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.214169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.214175 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.214767 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.214811 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"spinlock_wait_cycles":41856,"thread_start_us":92,"threads_started":1}
I20260924 20:13:32.214949 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.210794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:46753 (local address 127.0.0.1:32988)
0924 20:13:32.211650 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.211706 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.211874 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.213030 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.213033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.213048 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.213486 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.213489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.213501 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.213532 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.213954 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.213957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.213960 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.214751 (+ 791us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.214792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":106,"threads_started":1}
I20260924 20:13:32.218034 2307 tablet_service.cc:1467] Processing CreateTablet for tablet 8715249e93d04aeb8f3f670d1f76643a (DEFAULT_TABLE table=test-table [id=e5e930d9a7724884a73d5c10905821c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:32.218361 2438 tablet_service.cc:1467] Processing CreateTablet for tablet 8715249e93d04aeb8f3f670d1f76643a (DEFAULT_TABLE table=test-table [id=e5e930d9a7724884a73d5c10905821c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:32.219239 2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8715249e93d04aeb8f3f670d1f76643a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:32.219316 2438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8715249e93d04aeb8f3f670d1f76643a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:32.219148 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 8715249e93d04aeb8f3f670d1f76643a (DEFAULT_TABLE table=test-table [id=e5e930d9a7724884a73d5c10905821c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:32.220083 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8715249e93d04aeb8f3f670d1f76643a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:32.226343 2652 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Bootstrap starting.
I20260924 20:13:32.227006 2653 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Bootstrap starting.
I20260924 20:13:32.227532 2654 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Bootstrap starting.
I20260924 20:13:32.228622 2652 tablet_bootstrap.cc:654] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:32.229406 2653 tablet_bootstrap.cc:654] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:32.229444 2652 log.cc:824] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:32.229588 2654 tablet_bootstrap.cc:654] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:32.230419 2654 log.cc:824] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:32.230419 2653 log.cc:824] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:32.232092 2652 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: No bootstrap required, opened a new log
I20260924 20:13:32.232327 2652 ts_tablet_manager.cc:1397] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:13:32.232659 2653 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: No bootstrap required, opened a new log
I20260924 20:13:32.232863 2653 ts_tablet_manager.cc:1397] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:13:32.232983 2654 tablet_bootstrap.cc:492] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: No bootstrap required, opened a new log
I20260924 20:13:32.233165 2654 ts_tablet_manager.cc:1397] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 20:13:32.236171 2652 raft_consensus.cc:348] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.236635 2652 raft_consensus.cc:374] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:32.236698 2652 raft_consensus.cc:729] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078cb3d4c1b048c58e444c7857d2e5e1, State: Initialized, Role: FOLLOWER
I20260924 20:13:32.237221 2652 consensus_queue.cc:260] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.237709 2653 raft_consensus.cc:348] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.237843 2654 raft_consensus.cc:348] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.238045 2654 raft_consensus.cc:374] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:32.238044 2653 raft_consensus.cc:374] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:32.238116 2654 raft_consensus.cc:729] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7565c1d32565478cbda34c2c9e6d636c, State: Initialized, Role: FOLLOWER
I20260924 20:13:32.238116 2653 raft_consensus.cc:729] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a56ecf1b8d41a3b841e29b7a6996d8, State: Initialized, Role: FOLLOWER
I20260924 20:13:32.238396 2504 heartbeater.cc:503] Master 127.29.83.190:38627 was elected leader, sending a full tablet report...
I20260924 20:13:32.238466 2652 ts_tablet_manager.cc:1428] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:13:32.238626 2653 consensus_queue.cc:260] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.238626 2654 consensus_queue.cc:260] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.239557 2373 heartbeater.cc:503] Master 127.29.83.190:38627 was elected leader, sending a full tablet report...
I20260924 20:13:32.239646 2653 ts_tablet_manager.cc:1428] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:13:32.239646 2654 ts_tablet_manager.cc:1428] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:13:32.240664 2635 heartbeater.cc:503] Master 127.29.83.190:38627 was elected leader, sending a full tablet report...
W20260924 20:13:32.312288 2374 tablet.cc:2369] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:32.419412 2660 raft_consensus.cc:482] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:32.419665 2660 raft_consensus.cc:504] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.420751 2660 leader_election.cc:288] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7565c1d32565478cbda34c2c9e6d636c (127.29.83.131:34555), 078cb3d4c1b048c58e444c7857d2e5e1 (127.29.83.130:46753)
I20260924 20:13:32.423275 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.421103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:34555 (local address 127.29.83.129:57119)
0924 20:13:32.421490 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.421502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.421636 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.421931 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.421934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.421949 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.422479 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.422482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.422495 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.422530 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.423037 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.423040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.423042 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.423109 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.423158 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":118,"threads_started":1}
I20260924 20:13:32.423324 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.421208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:57119 (local address 127.29.83.131:34555)
0924 20:13:32.421336 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.421340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.421598 (+ 258us) server_negotiation.cc:407] Connection header received
0924 20:13:32.421747 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.421750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.421813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.421906 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.422675 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.422680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.422682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.422695 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.422713 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.422722 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.422821 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.423166 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.423208 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:13:32.423354 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.421103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:46753 (local address 127.29.83.129:42745)
0924 20:13:32.421669 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.421682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.421849 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.422137 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.422140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.422157 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:32.422684 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:13:32.422687 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:32.422697 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:13:32.422723 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:32.423026 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:32.423029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:32.423032 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:13:32.423162 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:13:32.423200 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":36,"thread_start_us":89,"threads_started":1}
I20260924 20:13:32.423631 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.421208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:42745 (local address 127.29.83.130:46753)
0924 20:13:32.421336 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.421341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.421789 (+ 448us) server_negotiation.cc:407] Connection header received
0924 20:13:32.421956 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.421959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.422004 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.422109 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.422867 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.422870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.422872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.422884 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.422903 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.422911 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.422996 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.423458 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.423507 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:32.424068 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8715249e93d04aeb8f3f670d1f76643a" candidate_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7565c1d32565478cbda34c2c9e6d636c" is_pre_election: true
I20260924 20:13:32.424369 2458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8715249e93d04aeb8f3f670d1f76643a" candidate_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" is_pre_election: true
I20260924 20:13:32.424413 2589 raft_consensus.cc:2454] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a56ecf1b8d41a3b841e29b7a6996d8 in term 0.
I20260924 20:13:32.424664 2458 raft_consensus.cc:2454] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a56ecf1b8d41a3b841e29b7a6996d8 in term 0.
I20260924 20:13:32.424906 2261 leader_election.cc:302] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7565c1d32565478cbda34c2c9e6d636c, a6a56ecf1b8d41a3b841e29b7a6996d8; no voters:
I20260924 20:13:32.425208 2660 raft_consensus.cc:2781] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:32.425323 2660 raft_consensus.cc:482] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:32.425369 2660 raft_consensus.cc:3037] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 20:13:32.425783 2636 tablet.cc:2369] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:32.426762 2660 raft_consensus.cc:504] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.427145 2660 leader_election.cc:288] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [CANDIDATE]: Term 1 election: Requested vote from peers 7565c1d32565478cbda34c2c9e6d636c (127.29.83.131:34555), 078cb3d4c1b048c58e444c7857d2e5e1 (127.29.83.130:46753)
I20260924 20:13:32.427479 2458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8715249e93d04aeb8f3f670d1f76643a" candidate_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078cb3d4c1b048c58e444c7857d2e5e1"
I20260924 20:13:32.427479 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8715249e93d04aeb8f3f670d1f76643a" candidate_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7565c1d32565478cbda34c2c9e6d636c"
I20260924 20:13:32.427603 2589 raft_consensus.cc:3037] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:32.427603 2458 raft_consensus.cc:3037] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:32.428810 2589 raft_consensus.cc:2454] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a56ecf1b8d41a3b841e29b7a6996d8 in term 1.
I20260924 20:13:32.428810 2458 raft_consensus.cc:2454] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a56ecf1b8d41a3b841e29b7a6996d8 in term 1.
I20260924 20:13:32.429082 2261 leader_election.cc:302] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7565c1d32565478cbda34c2c9e6d636c, a6a56ecf1b8d41a3b841e29b7a6996d8; no voters:
I20260924 20:13:32.429273 2660 raft_consensus.cc:2781] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:32.429507 2660 raft_consensus.cc:686] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [term 1 LEADER]: Becoming Leader. State: Replica: a6a56ecf1b8d41a3b841e29b7a6996d8, State: Running, Role: LEADER
I20260924 20:13:32.429890 2660 consensus_queue.cc:237] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } }
I20260924 20:13:32.433368 2185 catalog_manager.cc:5495] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a56ecf1b8d41a3b841e29b7a6996d8 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a56ecf1b8d41a3b841e29b7a6996d8" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34125 } health_report { overall_health: HEALTHY } } }
I20260924 20:13:32.455832 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.454281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.29.83.129:34125)
0924 20:13:32.454417 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.454422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.454436 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:32.454579 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.454582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.454646 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.454744 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.455406 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.455410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.455412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.455423 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.455451 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.455459 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.455549 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.455649 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.455693 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:13:32.462141 2458 raft_consensus.cc:1261] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 1 FOLLOWER]: Refusing update from remote peer a6a56ecf1b8d41a3b841e29b7a6996d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:32.462162 2589 raft_consensus.cc:1261] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 1 FOLLOWER]: Refusing update from remote peer a6a56ecf1b8d41a3b841e29b7a6996d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:32.462865 2663 consensus_queue.cc:1035] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078cb3d4c1b048c58e444c7857d2e5e1" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:32.463043 2660 consensus_queue.cc:1035] T 8715249e93d04aeb8f3f670d1f76643a P a6a56ecf1b8d41a3b841e29b7a6996d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7565c1d32565478cbda34c2c9e6d636c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:32.473896 2668 mvcc.cc:205] Tried to move back new op lower bound from 7332990207836999680 to 7332990207715540992. Current Snapshot: MvccSnapshot[applied={T|T < 7332990207836999680}]
I20260924 20:13:32.479511 2184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:32.479681 2184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:32.482187 2184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54dae85f8bf347a7898b3ac8de7f4e8b: Sending DeleteTablet for 3 replicas of tablet 8715249e93d04aeb8f3f670d1f76643a
I20260924 20:13:32.483083 2438 tablet_service.cc:1514] Processing DeleteTablet for tablet 8715249e93d04aeb8f3f670d1f76643a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:32 UTC) from {username='slave'} at 127.0.0.1:32988
I20260924 20:13:32.483083 2569 tablet_service.cc:1514] Processing DeleteTablet for tablet 8715249e93d04aeb8f3f670d1f76643a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:32 UTC) from {username='slave'} at 127.0.0.1:36288
I20260924 20:13:32.483508 2307 tablet_service.cc:1514] Processing DeleteTablet for tablet 8715249e93d04aeb8f3f670d1f76643a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:32 UTC) from {username='slave'} at 127.0.0.1:42720
I20260924 20:13:32.483597 2677 tablet_replica.cc:330] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: stopping tablet replica
I20260924 20:13:32.483597 2676 tablet_replica.cc:330] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: stopping tablet replica
I20260924 20:13:32.483923 2676 raft_consensus.cc:2229] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:32.483923 2677 raft_consensus.cc:2229] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:32.484256 2676 raft_consensus.cc:2258] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:32.484257 2677 raft_consensus.cc:2258] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:32.484759 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2245
I20260924 20:13:32.486650 2677 ts_tablet_manager.cc:1905] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:32.487421 2676 ts_tablet_manager.cc:1905] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:32.489046 2677 ts_tablet_manager.cc:1918] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:32.489164 2677 log.cc:1196] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/8715249e93d04aeb8f3f670d1f76643a
I20260924 20:13:32.489544 2677 ts_tablet_manager.cc:1939] T 8715249e93d04aeb8f3f670d1f76643a P 7565c1d32565478cbda34c2c9e6d636c: Deleting consensus metadata
I20260924 20:13:32.489673 2676 ts_tablet_manager.cc:1918] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:32.489825 2676 log.cc:1196] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/8715249e93d04aeb8f3f670d1f76643a
I20260924 20:13:32.490111 2676 ts_tablet_manager.cc:1939] T 8715249e93d04aeb8f3f670d1f76643a P 078cb3d4c1b048c58e444c7857d2e5e1: Deleting consensus metadata
I20260924 20:13:32.491080 2170 catalog_manager.cc:4841] TS 7565c1d32565478cbda34c2c9e6d636c (127.29.83.131:34555): tablet 8715249e93d04aeb8f3f670d1f76643a (table test-table [id=e5e930d9a7724884a73d5c10905821c0]) successfully deleted
I20260924 20:13:32.491148 2172 catalog_manager.cc:4841] TS 078cb3d4c1b048c58e444c7857d2e5e1 (127.29.83.130:46753): tablet 8715249e93d04aeb8f3f670d1f76643a (table test-table [id=e5e930d9a7724884a73d5c10905821c0]) successfully deleted
W20260924 20:13:32.495828 2172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:34125 (error 108)
I20260924 20:13:32.495919 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2376
I20260924 20:13:32.497429 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.497044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34125 (local address 127.0.0.1:42746)
0924 20:13:32.497273 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.497350 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:34125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260924 20:13:32.497646 2172 catalog_manager.cc:4584] TS a6a56ecf1b8d41a3b841e29b7a6996d8 (127.29.83.129:34125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8715249e93d04aeb8f3f670d1f76643a: Network error: Client connection negotiation failed: client connection to 127.29.83.129:34125: connect: Connection refused (error 111)
I20260924 20:13:32.501994 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2507
I20260924 20:13:32.507875 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2154
2026-09-24T20:13:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1138 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 20:13:32.530730 30030 mini_oidc.cc:152] Starting JWKS server
I20260924 20:13:32.532423 30030 webserver.cc:460] Webserver started at https://127.0.0.1:37521/ using document root <none> and password file <none>
I20260924 20:13:32.532640 30030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:13:32.532954 30030 webserver.cc:460] Webserver started at http://127.0.0.1:36953/ using document root <none> and password file <none>
2026-09-24T20:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:32Z Disabled control of system clock
I20260924 20:13:32.542611 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40347
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40347
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37521/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/testchainca.cert with env {}
W20260924 20:13:32.654961 2685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:32.655246 2685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:32.655287 2685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:32.659359 2685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:32.659416 2685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:32.659432 2685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:32.659448 2685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:32.659463 2685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:13:32.659479 2685 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:13:32.659492 2685 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37521/jwks.json
I20260924 20:13:32.664772 2685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37521/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:32.665794 2685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:32.666775 2685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:32.675293 2690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:32.675377 2685 server_base.cc:1034] running on GCE node
W20260924 20:13:32.675275 2693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:32.675272 2691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:32.676203 2685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:32.677352 2685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:32.678552 2685 hybrid_clock.cc:648] HybridClock initialized: now 1790280812678538 us; error 27 us; skew 500 ppm
I20260924 20:13:32.680447 2685 webserver.cc:460] Webserver started at http://127.29.83.190:33255/ using document root <none> and password file <none>
I20260924 20:13:32.681031 2685 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:32.681123 2685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:32.681344 2685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:32.683429 2685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "33838745d1df445d8a1605280e332f09"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "7c06ba4a041af523490b28f73882053d"
server_key_iv: "412cfc361678c4de3b482c518ebbf70e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.684003 2685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "33838745d1df445d8a1605280e332f09"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "7c06ba4a041af523490b28f73882053d"
server_key_iv: "412cfc361678c4de3b482c518ebbf70e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.687631 2685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 20:13:32.690198 2699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:32.691514 2685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:13:32.691677 2685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "33838745d1df445d8a1605280e332f09"
format_stamp: "Formatted at 2026-09-24 20:13:32 on dist-test-slave-52fr"
server_key: "7c06ba4a041af523490b28f73882053d"
server_key_iv: "412cfc361678c4de3b482c518ebbf70e"
server_key_version: "encryptionkey@0"
I20260924 20:13:32.691805 2685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:32.717087 2685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:32.717952 2685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:32.718149 2685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:32.725382 2685 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40347
I20260924 20:13:32.725386 2751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40347 every 8 connection(s)
I20260924 20:13:32.726477 2685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:32.728746 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2685
I20260924 20:13:32.728852 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:32.729130 30030 external_mini_cluster.cc:1428] Setting key 562c90602e30df09632102dd12a82f17
I20260924 20:13:32.730067 2752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:32.737082 2752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09: Bootstrap starting.
I20260924 20:13:32.738751 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.730592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.29.83.190:40347)
0924 20:13:32.731225 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.731241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.731275 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:13:32.731926 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.731943 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.732351 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.732683 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:32.733029 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.734115 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.735371 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.735884 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.736026 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:32.736879 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:32.736884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:32.736897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:32.737370 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:32.737427 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:32.737448 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:32.737523 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:32.737693 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 20:13:32.738006 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":135,"threads_started":1}
I20260924 20:13:32.739704 2752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:32.740502 2752 log.cc:824] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:32.741714 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:13:32.742671 2752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09: No bootstrap required, opened a new log
I20260924 20:13:32.746246 2752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } }
I20260924 20:13:32.746479 2752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:32.746515 2752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33838745d1df445d8a1605280e332f09, State: Initialized, Role: FOLLOWER
I20260924 20:13:32.746995 2752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } }
I20260924 20:13:32.747124 2752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:32.747208 2752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:32.747332 2752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:32.748293 2752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } }
I20260924 20:13:32.748641 2752 leader_election.cc:302] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33838745d1df445d8a1605280e332f09; no voters:
I20260924 20:13:32.749003 2752 leader_election.cc:288] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:32.749085 2757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:32.749301 2757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [term 1 LEADER]: Becoming Leader. State: Replica: 33838745d1df445d8a1605280e332f09, State: Running, Role: LEADER
I20260924 20:13:32.749605 2757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } }
I20260924 20:13:32.750367 2752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:32.751277 2759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33838745d1df445d8a1605280e332f09. Latest consensus state: current_term: 1 leader_uuid: "33838745d1df445d8a1605280e332f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } } }
I20260924 20:13:32.751545 2758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33838745d1df445d8a1605280e332f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33838745d1df445d8a1605280e332f09" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40347 } } }
I20260924 20:13:32.751642 2758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:32.751834 2759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:32.752045 2766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:32.755275 2766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:32.761224 2766 catalog_manager.cc:1347] Generated new cluster ID: 6dc95ce4eb1d40968e8b4226e5d7247c
I20260924 20:13:32.761315 2766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:32.774842 2766 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:32.776134 2766 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:32.784031 2766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33838745d1df445d8a1605280e332f09: Generated new TSK 0
I20260924 20:13:32.784868 2766 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:13:32.811435 2782 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 20:13:32.811591 2782 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 20:13:32.812215 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:32.795010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.29.83.190:40347)
0924 20:13:32.795146 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:32.795150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:32.795333 (+ 183us) server_negotiation.cc:407] Connection header received
0924 20:13:32.795585 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:32.795589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:32.795644 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:32.795755 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:13:32.795986 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.796846 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.797693 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.798052 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.798152 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:32.798457 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:13:32.811727 (+ 13270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:13:32.812034 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 20:13:32.812289 2755 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 20:13:32.813410 2781 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:32.794880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40347 (local address 127.0.0.1:50126)
0924 20:13:32.795251 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:32.795264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:32.795420 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:32.795775 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:32.795777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:32.795784 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:13:32.795882 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:32.795887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.796968 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.796971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:32.797587 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:32.797594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:32.798157 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:32.798160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:32.798239 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:32.798281 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:13:32.812026 (+ 13745us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:13:32.812256 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.83.190:40347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.83.190:40347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 20:13:32.814227 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2685
2026-09-24T20:13:32Z chronyd exiting
I20260924 20:13:32.852098 30030 test_util.cc:175] -----------------------------------------------
I20260924 20:13:32.852190 30030 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280784619584-30030-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 20:13:32.853266 30030 mini_oidc.cc:152] Starting JWKS server
I20260924 20:13:32.854251 30030 webserver.cc:460] Webserver started at https://127.0.0.1:34335/ using document root <none> and password file <none>
I20260924 20:13:32.854336 30030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:13:32.854606 30030 webserver.cc:460] Webserver started at http://127.0.0.1:39779/ using document root <none> and password file <none>
2026-09-24T20:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:32Z Disabled control of system clock
I20260924 20:13:32.864599 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35469
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:35469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34335/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/testchainca.cert with env {}
W20260924 20:13:32.978550 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:32.978858 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:32.978906 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:32.983093 2788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:32.983172 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:32.983189 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:32.983206 2788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:32.983222 2788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:13:32.983238 2788 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:13:32.983253 2788 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34335/jwks.json
I20260924 20:13:32.988647 2788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:35469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34335/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:32.989696 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:32.990779 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:32.999636 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:32.999703 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:32.999658 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:32.999955 2788 server_base.cc:1034] running on GCE node
I20260924 20:13:33.000524 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:33.001654 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:33.002827 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790280813002798 us; error 46 us; skew 500 ppm
I20260924 20:13:33.004709 2788 webserver.cc:460] Webserver started at http://127.29.83.190:35867/ using document root <none> and password file <none>
I20260924 20:13:33.005326 2788 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:33.005388 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:33.005651 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:33.007797 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "de3a3780885b4fec9775263b5d7560fe"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "2316def1f062016d62b9218872aaed68"
server_key_iv: "ef0ff937ffa50f341016283ce189ed91"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.008385 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "de3a3780885b4fec9775263b5d7560fe"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "2316def1f062016d62b9218872aaed68"
server_key_iv: "ef0ff937ffa50f341016283ce189ed91"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.012323 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:13:33.014992 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:33.016294 2788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:33.016435 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "de3a3780885b4fec9775263b5d7560fe"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "2316def1f062016d62b9218872aaed68"
server_key_iv: "ef0ff937ffa50f341016283ce189ed91"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.016546 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:33.032176 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:33.032922 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:33.033131 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:33.040442 2788 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:35469
I20260924 20:13:33.040472 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:35469 every 8 connection(s)
I20260924 20:13:33.041436 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:33.044683 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:33.050307 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe: Bootstrap starting.
I20260924 20:13:33.050917 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2788
I20260924 20:13:33.051074 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:33.051360 30030 external_mini_cluster.cc:1428] Setting key 093cf4dbda482b4748930ba25880c742
I20260924 20:13:33.053205 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:33.054021 2855 log.cc:824] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:33.056210 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe: No bootstrap required, opened a new log
I20260924 20:13:33.059715 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } }
I20260924 20:13:33.059991 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:33.060058 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3a3780885b4fec9775263b5d7560fe, State: Initialized, Role: FOLLOWER
I20260924 20:13:33.060499 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } }
I20260924 20:13:33.060626 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:33.060719 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:33.060863 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:33.061049 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.052664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.29.83.190:35469)
0924 20:13:33.053497 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.053513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.053549 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:13:33.054336 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.054356 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.054806 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.055186 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:33.055619 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.056627 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.057511 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.058064 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.058242 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.059079 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:33.059084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:33.059097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:33.059475 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:33.059529 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:33.059548 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:33.059658 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:33.059771 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 20:13:33.060064 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260924 20:13:33.061890 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } }
I20260924 20:13:33.062255 2855 leader_election.cc:302] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3a3780885b4fec9775263b5d7560fe; no voters:
I20260924 20:13:33.062605 2855 leader_election.cc:288] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:33.062768 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:33.063038 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [term 1 LEADER]: Becoming Leader. State: Replica: de3a3780885b4fec9775263b5d7560fe, State: Running, Role: LEADER
I20260924 20:13:33.063374 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } }
I20260924 20:13:33.063958 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:33.064100 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:13:33.065630 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3a3780885b4fec9775263b5d7560fe. Latest consensus state: current_term: 1 leader_uuid: "de3a3780885b4fec9775263b5d7560fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } } }
I20260924 20:13:33.065815 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.066200 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3a3780885b4fec9775263b5d7560fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3a3780885b4fec9775263b5d7560fe" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35469 } } }
I20260924 20:13:33.066473 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:33.066473 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.069947 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:33.075258 2874 catalog_manager.cc:1347] Generated new cluster ID: 020ac91b122b48f49ed57a7ba5000aca
I20260924 20:13:33.075344 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:33.095672 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:33.096643 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:33.102335 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3a3780885b4fec9775263b5d7560fe: Generated new TSK 0
I20260924 20:13:33.103077 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:13:33.143903 2885 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 20:13:33.144007 2885 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 20:13:33.144786 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.120956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.29.83.190:35469)
0924 20:13:33.121109 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.121114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.121389 (+ 275us) server_negotiation.cc:407] Connection header received
0924 20:13:33.121511 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.121514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.121562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.121653 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:13:33.121941 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.122491 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.123804 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.124059 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.124159 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.124515 (+ 356us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:13:33.144343 (+ 19828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:13:33.144652 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 20:13:33.144892 2858 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 20:13:33.144948 2884 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:33.120926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35469 (local address 127.0.0.1:59388)
0924 20:13:33.121257 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:33.121272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:33.121389 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:33.121676 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:33.121679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:33.121687 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:13:33.121823 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.121830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.122606 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.122610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.123616 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.123629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.124208 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.124213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.124303 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.124338 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:13:33.144579 (+ 20241us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:13:33.144766 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.83.190:35469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
I20260924 20:13:33.145553 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2788
2026-09-24T20:13:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 20:13:33.184613 30030 mini_oidc.cc:152] Starting JWKS server
I20260924 20:13:33.185578 30030 webserver.cc:460] Webserver started at https://127.0.0.1:45987/ using document root <none> and password file <none>
I20260924 20:13:33.185726 30030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:13:33.186060 30030 webserver.cc:460] Webserver started at http://127.0.0.1:40175/ using document root <none> and password file <none>
2026-09-24T20:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:33Z Disabled control of system clock
I20260924 20:13:33.195219 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:42853
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:42853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45987/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:13:33.309293 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:33.309612 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:33.309655 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:33.313943 2891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:33.314025 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:33.314043 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:33.314061 2891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:33.314078 2891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:13:33.314095 2891 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:13:33.314108 2891 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45987/jwks.json
I20260924 20:13:33.319200 2891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:42853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45987/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:33.320277 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:33.321736 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:33.330876 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:33.330865 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:33.330929 2891 server_base.cc:1034] running on GCE node
W20260924 20:13:33.330865 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:33.331773 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:33.333031 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:33.334237 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790280813334222 us; error 35 us; skew 500 ppm
I20260924 20:13:33.336782 2891 webserver.cc:460] Webserver started at http://127.29.83.190:40235/ using document root <none> and password file <none>
I20260924 20:13:33.337570 2891 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:33.337695 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:33.337973 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:33.341107 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "ebf6a4e746134e8bb4202091f6645a0a"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "f655d662b3305707212f0bd38cc3ef59"
server_key_iv: "924631fbe6ae2bd811a47841fd165755"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.341809 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "ebf6a4e746134e8bb4202091f6645a0a"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "f655d662b3305707212f0bd38cc3ef59"
server_key_iv: "924631fbe6ae2bd811a47841fd165755"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.345971 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:13:33.348702 2905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:33.350157 2891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:33.350318 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "ebf6a4e746134e8bb4202091f6645a0a"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "f655d662b3305707212f0bd38cc3ef59"
server_key_iv: "924631fbe6ae2bd811a47841fd165755"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.350448 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:33.380399 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:33.381170 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:33.381378 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:33.388507 2891 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:42853
I20260924 20:13:33.388528 2957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:42853 every 8 connection(s)
I20260924 20:13:33.389475 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:33.391975 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2891
I20260924 20:13:33.392114 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:33.392408 30030 external_mini_cluster.cc:1428] Setting key dc7ffc48991a7d2d0b0521f9a6e9c573
I20260924 20:13:33.393033 2958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:33.400362 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a: Bootstrap starting.
I20260924 20:13:33.402128 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.393901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.29.83.190:42853)
0924 20:13:33.394550 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.394566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.394601 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:13:33.395274 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.395291 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.395685 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.395983 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:33.396478 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.397464 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.398706 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.399203 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.399325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.400240 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:33.400245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:33.400258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:33.400707 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:33.400762 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:33.400782 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:33.400858 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:33.401027 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 20:13:33.401343 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":145,"threads_started":1}
I20260924 20:13:33.402936 2958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:33.403863 2958 log.cc:824] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:33.405339 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:13:33.406374 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a: No bootstrap required, opened a new log
I20260924 20:13:33.410329 2958 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } }
I20260924 20:13:33.410604 2958 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:33.410701 2958 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf6a4e746134e8bb4202091f6645a0a, State: Initialized, Role: FOLLOWER
I20260924 20:13:33.411198 2958 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } }
I20260924 20:13:33.411353 2958 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:33.411442 2958 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:33.411546 2958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:33.412616 2958 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } }
I20260924 20:13:33.412969 2958 leader_election.cc:302] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf6a4e746134e8bb4202091f6645a0a; no voters:
I20260924 20:13:33.413344 2958 leader_election.cc:288] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:33.413566 2963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:33.414211 2963 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [term 1 LEADER]: Becoming Leader. State: Replica: ebf6a4e746134e8bb4202091f6645a0a, State: Running, Role: LEADER
I20260924 20:13:33.414520 2963 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } }
I20260924 20:13:33.414748 2958 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:33.416306 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf6a4e746134e8bb4202091f6645a0a. Latest consensus state: current_term: 1 leader_uuid: "ebf6a4e746134e8bb4202091f6645a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } } }
I20260924 20:13:33.416368 2964 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf6a4e746134e8bb4202091f6645a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6a4e746134e8bb4202091f6645a0a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42853 } } }
I20260924 20:13:33.416467 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.416473 2964 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.416904 2976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:33.419771 2976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:33.425158 2976 catalog_manager.cc:1347] Generated new cluster ID: 6a4129dff53944269b960765a4d0f361
I20260924 20:13:33.425236 2976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:33.438213 2976 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:33.439512 2976 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:33.450664 2976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf6a4e746134e8bb4202091f6645a0a: Generated new TSK 0
I20260924 20:13:33.451578 2976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:13:33.472014 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.466578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.29.83.190:42853)
0924 20:13:33.466739 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.466743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.466918 (+ 175us) server_negotiation.cc:407] Connection header received
0924 20:13:33.467014 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.467016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.467068 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.467166 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:13:33.467443 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.468015 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.469306 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.469545 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.469629 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.470042 (+ 413us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:13:33.471632 (+ 1590us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45987/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:13:33.471869 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45987/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 20:13:33.472102 2961 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45987/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 20:13:33.473908 2987 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:33.466414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:42853 (local address 127.0.0.1:50234)
0924 20:13:33.466757 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:33.466775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:33.466901 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:33.467194 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:33.467197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:33.467205 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:13:33.467314 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.467321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.468138 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.468142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.469162 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.469173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.469674 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.469678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.469788 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.469819 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:13:33.471807 (+ 1988us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45987/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:13:33.473690 (+ 1883us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.83.190:42853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45987/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1643,"thread_start_us":111,"threads_started":1}
I20260924 20:13:33.474458 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2891
E20260924 20:13:33.485492 2988 webserver.cc:533] Webserver: sslize failed:
E20260924 20:13:33.485607 2988 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T20:13:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T20:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:33Z Disabled control of system clock
I20260924 20:13:33.524988 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36023
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36023
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/testchainca.cert with env {}
W20260924 20:13:33.642390 2992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:33.642715 2992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:33.642819 2992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:33.647071 2992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:33.647146 2992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:33.647162 2992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:33.647181 2992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:33.647197 2992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:13:33.647253 2992 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:13:33.647279 2992 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 20:13:33.652933 2992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:33.654170 2992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:33.655229 2992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:33.664433 2997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:33.664492 2998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:33.664587 2992 server_base.cc:1034] running on GCE node
W20260924 20:13:33.664815 3000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:33.665442 2992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:33.666735 2992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:33.667940 2992 hybrid_clock.cc:648] HybridClock initialized: now 1790280813667911 us; error 45 us; skew 500 ppm
I20260924 20:13:33.670009 2992 webserver.cc:460] Webserver started at http://127.29.83.190:38051/ using document root <none> and password file <none>
I20260924 20:13:33.670630 2992 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:33.670729 2992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:33.670961 2992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:33.673066 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "c7763ac2946c480a95ff44c58ec77255"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "678d9304b964f0fb0b00a68ad530679b"
server_key_iv: "d3ee26627dd77e1fe5e44ea7a93ed772"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.673666 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "c7763ac2946c480a95ff44c58ec77255"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "678d9304b964f0fb0b00a68ad530679b"
server_key_iv: "d3ee26627dd77e1fe5e44ea7a93ed772"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.677397 2992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:33.680106 3006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:33.681494 2992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:33.681666 2992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "c7763ac2946c480a95ff44c58ec77255"
format_stamp: "Formatted at 2026-09-24 20:13:33 on dist-test-slave-52fr"
server_key: "678d9304b964f0fb0b00a68ad530679b"
server_key_iv: "d3ee26627dd77e1fe5e44ea7a93ed772"
server_key_version: "encryptionkey@0"
I20260924 20:13:33.681879 2992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:33.694828 2992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:33.695537 2992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:33.695757 2992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:33.702847 2992 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36023
I20260924 20:13:33.702848 3058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36023 every 8 connection(s)
I20260924 20:13:33.703855 2992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:33.707191 3059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:33.711712 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 2992
I20260924 20:13:33.711853 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:33.712131 30030 external_mini_cluster.cc:1428] Setting key 4da7b92e934edad1212a8ca0ff1a4db1
I20260924 20:13:33.713676 3059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255: Bootstrap starting.
I20260924 20:13:33.716159 3059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:33.716935 3059 log.cc:824] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:33.719594 3059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255: No bootstrap required, opened a new log
I20260924 20:13:33.722533 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.713589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.29.83.190:36023)
0924 20:13:33.714304 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.714325 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.714363 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:13:33.715022 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.715040 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.715527 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.715895 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:33.716415 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.717572 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.718995 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.719504 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.719638 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.720538 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:33.720544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:33.720557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:13:33.720967 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:33.721021 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:13:33.721050 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:33.721152 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:33.721325 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:13:33.721728 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":147,"threads_started":1}
I20260924 20:13:33.723783 3059 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } }
I20260924 20:13:33.724131 3059 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:33.724193 3059 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7763ac2946c480a95ff44c58ec77255, State: Initialized, Role: FOLLOWER
I20260924 20:13:33.724757 3059 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } }
I20260924 20:13:33.724923 3059 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:33.724994 3059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:33.725117 3059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:33.725905 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:13:33.726320 3059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } }
I20260924 20:13:33.726691 3059 leader_election.cc:302] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7763ac2946c480a95ff44c58ec77255; no voters:
I20260924 20:13:33.727051 3059 leader_election.cc:288] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:33.727212 3064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:33.727445 3064 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [term 1 LEADER]: Becoming Leader. State: Replica: c7763ac2946c480a95ff44c58ec77255, State: Running, Role: LEADER
I20260924 20:13:33.727761 3064 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } }
I20260924 20:13:33.728413 3059 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:33.728955 3065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7763ac2946c480a95ff44c58ec77255. Latest consensus state: current_term: 1 leader_uuid: "c7763ac2946c480a95ff44c58ec77255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } } }
I20260924 20:13:33.729096 3065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.729633 3066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7763ac2946c480a95ff44c58ec77255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7763ac2946c480a95ff44c58ec77255" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36023 } } }
I20260924 20:13:33.729739 3066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:33.730063 3069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:33.733326 3069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:33.739058 3069 catalog_manager.cc:1347] Generated new cluster ID: 85392d2ea5fa42b5a4e1c8a784404b1a
I20260924 20:13:33.739130 3069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:33.768036 3069 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:33.769410 3069 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:33.778573 3069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7763ac2946c480a95ff44c58ec77255: Generated new TSK 0
I20260924 20:13:33.779455 3069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:13:33.888516 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:33.797869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.29.83.190:36023)
0924 20:13:33.798018 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:33.798023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:33.798156 (+ 133us) server_negotiation.cc:407] Connection header received
0924 20:13:33.798384 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:33.798387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:33.798439 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:33.798556 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:13:33.798818 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.799385 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.800286 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.800523 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.800636 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.800974 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:13:33.888000 (+ 87026us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:13:33.888355 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 20:13:33.888623 3062 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 20:13:33.888644 3088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:33.797716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36023 (local address 127.0.0.1:37138)
0924 20:13:33.798049 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:33.798071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:33.798212 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:33.798564 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:33.798567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:33.798575 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:13:33.798704 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.798710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.799517 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.799524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.800159 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:33.800166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:33.800649 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:33.800653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:33.800758 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:33.800780 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:13:33.888255 (+ 87475us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:13:33.888454 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.83.190:36023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":145,"threads_started":1}
I20260924 20:13:33.889372 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 2992
2026-09-24T20:13:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (393 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:34 dist-test-slave-52fr krb5kdc[3100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:34 dist-test-slave-52fr krb5kdc[3100](info): set up 2 sockets
Sep 24 20:13:34 dist-test-slave-52fr krb5kdc[3100](info): commencing operation
krb5kdc: starting...
W20260924 20:13:35.190168 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:35.316357 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:42209
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:35.436647 3113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:35.437011 3113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:35.437101 3113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:35.441336 3113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:35.441433 3113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:35.441473 3113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:35.441533 3113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:35.441572 3113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:35.447305 3113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:35.448455 3113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:35.449568 3113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:35.458878 3119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:35.458882 3121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:35.458897 3118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:35.459173 3113 server_base.cc:1034] running on GCE node
I20260924 20:13:35.459882 3113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:35.461000 3113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:35.462205 3113 hybrid_clock.cc:648] HybridClock initialized: now 1790280815462181 us; error 35 us; skew 500 ppm
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:35.464979 3113 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:35.465947 3113 webserver.cc:460] Webserver started at http://127.29.83.190:35359/ using document root <none> and password file <none>
I20260924 20:13:35.466511 3113 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:35.466568 3113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:35.466741 3113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:35.468731 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "57738e54af064a7bb015ff1893deb91c"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "9ee60fefc6dbc54117e3c1be5d181340"
server_key_iv: "fa383ee45755f0ce25208973ebc05625"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.469270 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "57738e54af064a7bb015ff1893deb91c"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "9ee60fefc6dbc54117e3c1be5d181340"
server_key_iv: "fa383ee45755f0ce25208973ebc05625"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.473157 3113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:13:35.475843 3128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:35.477181 3113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:35.477324 3113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "57738e54af064a7bb015ff1893deb91c"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "9ee60fefc6dbc54117e3c1be5d181340"
server_key_iv: "fa383ee45755f0ce25208973ebc05625"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.477451 3113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:35.491101 3113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:35.494849 3113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:35.495070 3113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:35.502635 3113 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:42209
I20260924 20:13:35.502655 3180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:42209 every 8 connection(s)
I20260924 20:13:35.503635 3113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:35.506708 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:35.512418 3181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: Bootstrap starting.
I20260924 20:13:35.513288 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3113
I20260924 20:13:35.513417 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:35.513706 30030 external_mini_cluster.cc:1428] Setting key b4cc25c5ecf1ef6b3dc9eb947732396a
I20260924 20:13:35.515455 3181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:35.516212 3181 log.cc:824] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:35.518937 3181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: No bootstrap required, opened a new log
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:35.523713 3181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } }
I20260924 20:13:35.523967 3181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:35.524048 3181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57738e54af064a7bb015ff1893deb91c, State: Initialized, Role: FOLLOWER
I20260924 20:13:35.524495 3181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } }
I20260924 20:13:35.524646 3181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:35.524731 3181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:35.524854 3181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:35.525800 3181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } }
I20260924 20:13:35.526152 3181 leader_election.cc:302] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57738e54af064a7bb015ff1893deb91c; no voters:
I20260924 20:13:35.526468 3181 leader_election.cc:288] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:35.526576 3186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:35.526798 3186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [term 1 LEADER]: Becoming Leader. State: Replica: 57738e54af064a7bb015ff1893deb91c, State: Running, Role: LEADER
I20260924 20:13:35.527113 3186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } }
I20260924 20:13:35.527834 3181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:35.529790 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:35.514913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.29.83.190:42209)
0924 20:13:35.515395 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:35.515410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:35.515446 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:13:35.516152 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:35.516173 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:35.516861 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:35.517257 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:35.518209 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.519297 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.520457 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.520980 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.521130 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:35.524166 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:35.524173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:35.524195 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:35.524225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:35.526307 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:35.526823 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:35.526829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:35.526835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:35.526976 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:35.527304 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:35.527308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:35.527310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:35.527814 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:35.527997 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:35.528339 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 20:13:35.528686 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":139,"threads_started":1}
I20260924 20:13:35.530309 3187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57738e54af064a7bb015ff1893deb91c. Latest consensus state: current_term: 1 leader_uuid: "57738e54af064a7bb015ff1893deb91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } } }
I20260924 20:13:35.530522 3187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:35.530853 3188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57738e54af064a7bb015ff1893deb91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57738e54af064a7bb015ff1893deb91c" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 42209 } } }
I20260924 20:13:35.530942 3188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c [sys.catalog]: This master's current role is: LEADER
W20260924 20:13:35.533833 3200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:35.533914 3200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:35.534081 3201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:35.535097 3201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:35.540987 3201 catalog_manager.cc:1347] Generated new cluster ID: 996a81a2631048f4a6be6f0ba620a742
I20260924 20:13:35.541065 3201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:35.572242 3201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:35.573309 3201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:35.584626 3201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57738e54af064a7bb015ff1893deb91c: Generated new TSK 0
I20260924 20:13:35.585645 3201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:35.621734 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:35.741598 3207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:35.741873 3207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:35.741916 3207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:35.745898 3207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:35.745955 3207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:35.746047 3207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:35.751457 3207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:35.752501 3207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:35.753516 3207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:35.763098 3212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:35.763114 3213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:35.763093 3215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:35.763469 3207 server_base.cc:1034] running on GCE node
I20260924 20:13:35.763873 3207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:35.764515 3207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:35.765671 3207 hybrid_clock.cc:648] HybridClock initialized: now 1790280815765633 us; error 51 us; skew 500 ppm
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:35.768323 3207 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:35.769093 3207 webserver.cc:460] Webserver started at http://127.29.83.129:37327/ using document root <none> and password file <none>
I20260924 20:13:35.769603 3207 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:35.769658 3207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:35.769903 3207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:35.771817 3207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "bd663f262e244ad19c2ff505444c61b7"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "8a30fa2ce2319df31e2c23db31b69030"
server_key_iv: "c1ced5290131df53b18af5de1757cc5a"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.772315 3207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "bd663f262e244ad19c2ff505444c61b7"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "8a30fa2ce2319df31e2c23db31b69030"
server_key_iv: "c1ced5290131df53b18af5de1757cc5a"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.775935 3207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:13:35.778374 3222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:35.779627 3207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:35.779757 3207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "bd663f262e244ad19c2ff505444c61b7"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "8a30fa2ce2319df31e2c23db31b69030"
server_key_iv: "c1ced5290131df53b18af5de1757cc5a"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.779862 3207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:35.794648 3207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:35.798007 3207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:35.798223 3207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:35.798806 3207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:35.799641 3207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:35.799690 3207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:35.799755 3207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:35.799785 3207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:35.808145 3207 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:35467
I20260924 20:13:35.808172 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:35467 every 8 connection(s)
I20260924 20:13:35.809015 3207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:35.818938 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3207
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:35.819104 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:35.819401 30030 external_mini_cluster.cc:1428] Setting key a01ad006c81bb7d9340609f11b9cba1a
I20260924 20:13:35.824002 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:35.810896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:51233 (local address 127.29.83.190:42209)
0924 20:13:35.811032 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:35.811042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:35.811924 (+ 882us) server_negotiation.cc:407] Connection header received
0924 20:13:35.813265 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:35.813269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:35.813318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:35.813399 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:35.814952 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.815508 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.816725 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.816983 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.817078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:35.819770 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:35.819774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:35.819777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:35.819808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:35.821630 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:35.822240 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:35.822247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:35.822252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:35.822352 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:35.822723 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:35.822729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:35.822734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:35.822979 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:35.823059 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:35.823775 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 20:13:35.823883 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:35.825248 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:35.811217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:42209 (local address 127.29.83.129:51233)
0924 20:13:35.811742 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:35.811788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:35.813038 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:35.813716 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:35.813724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:35.814240 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:35.814763 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:35.814778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.815658 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.815662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:35.816578 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:35.816587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:35.817155 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:35.817159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:35.817306 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:35.817994 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:35.818010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:35.819552 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:35.821809 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:35.821822 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:35.821835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:35.822127 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:35.822458 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:35.822460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:35.822461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:35.822606 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:35.823114 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:35.823122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:35.823472 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 20:13:35.823697 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 20:13:35.824116 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260924 20:13:35.826538 3336 heartbeater.cc:347] Connected to a master server at 127.29.83.190:42209
I20260924 20:13:35.826803 3336 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:35.827329 3336 heartbeater.cc:511] Master 127.29.83.190:42209 requested a full tablet report, sending...
I20260924 20:13:35.828949 3145 ts_manager.cc:194] Registered new tserver with Master: bd663f262e244ad19c2ff505444c61b7 (127.29.83.129:35467)
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:35.830719 3145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:51233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:35.850199 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:35.961035 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:35.961300 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:35.961401 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:35.965528 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:35.965633 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:35.965740 3341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:35.971226 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:35.972393 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:35.973464 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:35.983422 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:35.983508 3341 server_base.cc:1034] running on GCE node
W20260924 20:13:35.983482 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:35.983435 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:35.984133 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:35.984838 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:35.986047 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790280815986033 us; error 33 us; skew 500 ppm
Sep 24 20:13:35 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:35.988719 3341 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:35.989568 3341 webserver.cc:460] Webserver started at http://127.29.83.130:38631/ using document root <none> and password file <none>
I20260924 20:13:35.990172 3341 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:35.990278 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:35.990504 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:35.992501 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "6ffd1965969e4e5e86cb5bccb561c34b"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "ffdc8b222a028c9e44ceb3563a169039"
server_key_iv: "d027902471274de35691e788a4a0a29b"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.993053 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "6ffd1965969e4e5e86cb5bccb561c34b"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "ffdc8b222a028c9e44ceb3563a169039"
server_key_iv: "d027902471274de35691e788a4a0a29b"
server_key_version: "encryptionkey@0"
I20260924 20:13:35.996814 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:13:35.999512 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.000748 3341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:36.000893 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "6ffd1965969e4e5e86cb5bccb561c34b"
format_stamp: "Formatted at 2026-09-24 20:13:35 on dist-test-slave-52fr"
server_key: "ffdc8b222a028c9e44ceb3563a169039"
server_key_iv: "d027902471274de35691e788a4a0a29b"
server_key_version: "encryptionkey@0"
I20260924 20:13:36.001016 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:36.021950 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:36.025063 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:36.025277 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:36.025939 3341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:36.026831 3341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:36.026899 3341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.026965 3341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:36.027015 3341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.036507 3341 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:40715
I20260924 20:13:36.036526 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:40715 every 8 connection(s)
I20260924 20:13:36.037433 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:36.046778 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3341
I20260924 20:13:36.046911 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:36.047206 30030 external_mini_cluster.cc:1428] Setting key d5f6a1080028a6b46ee4997c103cba13
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:36.052434 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.039310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:39145 (local address 127.29.83.190:42209)
0924 20:13:36.039467 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.039471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.040342 (+ 871us) server_negotiation.cc:407] Connection header received
0924 20:13:36.041849 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.041852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.041909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.041989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.043220 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.043766 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.044946 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.045184 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.045299 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.048175 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:36.048178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:36.048180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:36.048206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:36.050153 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.050807 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.050810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.050811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.050906 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.051296 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.051300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.051301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.051496 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:36.051604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:36.052130 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 20:13:36.052271 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:13:36.053402 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.039697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:42209 (local address 127.29.83.130:39145)
0924 20:13:36.040170 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:36.040215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:36.041595 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:36.042167 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:36.042175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:36.042564 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:36.043069 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:36.043084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.043902 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.043905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:36.044815 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:36.044825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.045343 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.045347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:36.045535 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.046220 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:36.046237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:36.048001 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:36.050294 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:36.050300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:36.050314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:36.050594 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:36.051011 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:36.051014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:36.051017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:36.051182 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:36.051603 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:36.051610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:36.051882 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 20:13:36.052087 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 20:13:36.052452 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260924 20:13:36.054853 3470 heartbeater.cc:347] Connected to a master server at 127.29.83.190:42209
I20260924 20:13:36.055115 3470 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:36.055683 3470 heartbeater.cc:511] Master 127.29.83.190:42209 requested a full tablet report, sending...
I20260924 20:13:36.056800 3145 ts_manager.cc:194] Registered new tserver with Master: 6ffd1965969e4e5e86cb5bccb561c34b (127.29.83.130:40715)
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:36.057657 3145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:39145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:36.076624 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:36.190183 3475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:36.190423 3475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:36.190464 3475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:36.194355 3475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:36.194401 3475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:36.194458 3475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:36.199507 3475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:36.200444 3475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:36.201393 3475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:36.210947 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:36.210929 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:36.210929 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:36.210987 3475 server_base.cc:1034] running on GCE node
I20260924 20:13:36.211498 3475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:36.212098 3475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:36.213279 3475 hybrid_clock.cc:648] HybridClock initialized: now 1790280816213271 us; error 55 us; skew 500 ppm
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.215924 3475 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:36.216754 3475 webserver.cc:460] Webserver started at http://127.29.83.131:46733/ using document root <none> and password file <none>
I20260924 20:13:36.217242 3475 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:36.217298 3475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:36.217449 3475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:36.219444 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "34a9e853861a4d569c6d590637336a0c"
format_stamp: "Formatted at 2026-09-24 20:13:36 on dist-test-slave-52fr"
server_key: "911a4650e5673f991ad24438610e6e80"
server_key_iv: "840d1e6763d87a6d48e4f9666644acde"
server_key_version: "encryptionkey@0"
I20260924 20:13:36.219936 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "34a9e853861a4d569c6d590637336a0c"
format_stamp: "Formatted at 2026-09-24 20:13:36 on dist-test-slave-52fr"
server_key: "911a4650e5673f991ad24438610e6e80"
server_key_iv: "840d1e6763d87a6d48e4f9666644acde"
server_key_version: "encryptionkey@0"
I20260924 20:13:36.223729 3475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 20:13:36.226186 3490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.227530 3475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:36.227670 3475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "34a9e853861a4d569c6d590637336a0c"
format_stamp: "Formatted at 2026-09-24 20:13:36 on dist-test-slave-52fr"
server_key: "911a4650e5673f991ad24438610e6e80"
server_key_iv: "840d1e6763d87a6d48e4f9666644acde"
server_key_version: "encryptionkey@0"
I20260924 20:13:36.227766 3475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:36.256178 3475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:36.259413 3475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:36.259563 3475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:36.260113 3475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:36.260885 3475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:36.260931 3475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.260972 3475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:36.260986 3475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:36.269975 3475 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:43513
I20260924 20:13:36.270002 3603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:43513 every 8 connection(s)
I20260924 20:13:36.270865 3475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:36.273569 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3475
I20260924 20:13:36.273696 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:36.274045 30030 external_mini_cluster.cc:1428] Setting key bb306c7acf4d15b330f86e124b2444aa
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:36.284601 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.272632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:36865 (local address 127.29.83.190:42209)
0924 20:13:36.272815 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.272819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.273801 (+ 982us) server_negotiation.cc:407] Connection header received
0924 20:13:36.275362 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.275366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.275420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.275523 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.276698 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.277226 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.278405 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.278625 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.278734 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.281128 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:36.281132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:36.281139 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:36.281161 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:36.282556 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.283081 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.283083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.283085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.283156 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.283483 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.283488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.283489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.283714 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:36.283817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:36.284321 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 20:13:36.284436 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 20:13:36.285419 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.273032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:42209 (local address 127.29.83.131:36865)
0924 20:13:36.273562 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:36.273621 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:36.275132 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:36.275693 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:36.275699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:36.276055 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:36.276564 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:36.276578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.277346 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.277354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:36.278288 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:36.278296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.278773 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.278777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:36.278938 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.279516 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:36.279532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:36.280976 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:36.282668 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:36.282674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:36.282688 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:36.282969 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:36.283255 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:36.283259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:36.283261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:36.283390 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:36.283813 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:36.283819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:36.284077 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 20:13:36.284281 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 20:13:36.284575 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260924 20:13:36.286746 3604 heartbeater.cc:347] Connected to a master server at 127.29.83.190:42209
I20260924 20:13:36.286978 3604 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:36.287477 3604 heartbeater.cc:511] Master 127.29.83.190:42209 requested a full tablet report, sending...
I20260924 20:13:36.288470 3145 ts_manager.cc:194] Registered new tserver with Master: 34a9e853861a4d569c6d590637336a0c (127.29.83.131:43513)
I20260924 20:13:36.289029 3145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:36865
I20260924 20:13:36.299132 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/test-admin.keytab.
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.313624 30030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 20:13:36.316361 3611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:36.315011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.29.83.129:35467)
0924 20:13:36.315282 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.315296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.315320 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:13:36.315405 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.315410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.315588 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.315729 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.316186 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47858: BlockingRecv error: recv got EOF from 127.0.0.1:47858 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
W20260924 20:13:36.316430 30030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.317417 30030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.318329 30030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:36.327811 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.319119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.29.83.129:35467)
0924 20:13:36.319231 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.319235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.319364 (+ 129us) server_negotiation.cc:407] Connection header received
0924 20:13:36.319446 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.319448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.319482 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.319545 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.320007 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.320756 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.321529 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.321972 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.322053 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.324322 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:36.324331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:36.324343 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:36.324366 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:36.325996 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.326436 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.326442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.326443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.326515 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.326795 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.326800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.326802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.327049 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:36.327264 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:36.327489 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 20:13:36.327688 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 20:13:36.332121 3611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:36.331079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.29.83.129:35467)
0924 20:13:36.331236 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.331239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.331268 (+ 29us) server_negotiation.cc:407] Connection header received
0924 20:13:36.331457 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.331460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.331507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.331581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.332007 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47880: BlockingRecv error: recv got EOF from 127.0.0.1:47880 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 20:13:36.332195 30030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.333292 30030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.334208 30030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:36.342403 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.335064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.29.83.129:35467)
0924 20:13:36.335177 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.335186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.335370 (+ 184us) server_negotiation.cc:407] Connection header received
0924 20:13:36.335435 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.335437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.335471 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.335528 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.335955 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.336468 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.337196 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.337420 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.337499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.339614 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:36.339617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:36.339619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:36.339639 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:36.340864 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.341236 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.341239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.341240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.341307 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.341524 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.341526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.341527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.341693 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:36.341857 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:36.342117 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 20:13:36.342209 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:13:36.344990 3611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:36.344104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.29.83.129:35467)
0924 20:13:36.344217 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.344221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.344367 (+ 146us) server_negotiation.cc:407] Connection header received
0924 20:13:36.344426 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.344428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.344461 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.344525 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.344890 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47898: BlockingRecv error: recv got EOF from 127.0.0.1:47898 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:13:36.345172 30030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280784619584-30030-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.346127 30030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:36.346958 30030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:13:36.354794 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:36.347671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.29.83.129:35467)
0924 20:13:36.347803 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:36.347806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:36.347818 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:36.347911 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:36.347913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:36.347945 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:36.348007 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:36.348422 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.348904 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.349613 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:36.349883 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:36.349960 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:36.351921 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:36.351923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:36.351925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:36.351945 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:36.353113 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.353519 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.353522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.353523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.353604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:36.353988 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:36.353991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:36.353992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:36.354162 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:36.354289 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:36.354496 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 20:13:36.354614 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:13:36.356089 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3207
I20260924 20:13:36.361836 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3341
I20260924 20:13:36.367435 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3475
I20260924 20:13:36.372726 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3113
2026-09-24T20:13:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2229 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3624](info): set up 2 sockets
Sep 24 20:13:36 dist-test-slave-52fr krb5kdc[3624](info): commencing operation
krb5kdc: starting...
W20260924 20:13:37.402163 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:37 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab.
Entry for principal oryx/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab.
I20260924 20:13:37.525398 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41729
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab
--principal=oryx/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:37.638373 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:37.638676 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:37.638720 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:37.642774 3637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:37.642848 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:37.642868 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:37.642885 3637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:37.642900 3637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:37.648408 3637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab
--principal=oryx/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:37.649515 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:37.650631 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:37.659476 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:37.659595 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:37.659479 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:37.659742 3637 server_base.cc:1034] running on GCE node
I20260924 20:13:37.660446 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:37.661604 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:37.662876 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790280817662863 us; error 29 us; skew 500 ppm
Sep 24 20:13:37 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:37.670132 3637 init.cc:377] Logged in from keytab as oryx/127.29.83.190@KRBTEST.COM (short username oryx)
I20260924 20:13:37.671191 3637 webserver.cc:460] Webserver started at http://127.29.83.190:40319/ using document root <none> and password file <none>
I20260924 20:13:37.671769 3637 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:37.671835 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:37.672008 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:37.674257 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "94683b42ca209c9b808ffa696abbd76a"
server_key_iv: "63e72aec83b7632daf320b13e9cd7ab0"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.674829 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "94683b42ca209c9b808ffa696abbd76a"
server_key_iv: "63e72aec83b7632daf320b13e9cd7ab0"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.678890 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:13:37.681650 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:37.683020 3637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:13:37.683166 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "94683b42ca209c9b808ffa696abbd76a"
server_key_iv: "63e72aec83b7632daf320b13e9cd7ab0"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.683314 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:37.708695 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:37.712028 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:37.712248 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:37.719920 3637 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:41729
I20260924 20:13:37.720242 3704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:41729 every 8 connection(s)
I20260924 20:13:37.720957 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:37.722052 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3637
I20260924 20:13:37.722218 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:37.722515 30030 external_mini_cluster.cc:1428] Setting key be421168e00ab6b1aaa5d0434091fd40
I20260924 20:13:37.724462 3705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:37.730195 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Bootstrap starting.
Sep 24 20:13:37 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:37.732530 3705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:37.733266 3705 log.cc:824] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:37.735289 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: No bootstrap required, opened a new log
I20260924 20:13:37.736850 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:37.723895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.29.83.190:41729)
0924 20:13:37.724326 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:37.724341 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:37.724376 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:13:37.725087 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:37.725105 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:37.725435 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:37.725812 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:37.726602 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:37.727590 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:37.728459 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:37.728935 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:37.729106 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:37.731848 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:37.731853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:37.731868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:37.731898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:37.733907 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:37.734358 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:37.734364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:37.734377 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:37.734491 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:37.734784 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:37.734788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:37.734790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:37.735207 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:37.735415 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:37.735704 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 20:13:37.736035 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":151,"threads_started":1}
I20260924 20:13:37.740345 3705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } }
I20260924 20:13:37.740603 3705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:37.740698 3705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3d9bbbceb34ec49ed1ae6eb8be46c4, State: Initialized, Role: FOLLOWER
I20260924 20:13:37.741138 3705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } }
I20260924 20:13:37.741294 3705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:37.741380 3705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:37.741503 3705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:37.742522 3705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } }
I20260924 20:13:37.742974 3705 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3d9bbbceb34ec49ed1ae6eb8be46c4; no voters:
I20260924 20:13:37.743393 3705 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:37.743515 3710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:37.743798 3710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3d9bbbceb34ec49ed1ae6eb8be46c4, State: Running, Role: LEADER
I20260924 20:13:37.744189 3710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } }
I20260924 20:13:37.744925 3705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:37.745982 3712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } }
I20260924 20:13:37.746110 3712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:37.746440 3711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } }
I20260924 20:13:37.746537 3711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:37.746621 3719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:37.749935 3719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:37.755976 3719 catalog_manager.cc:1347] Generated new cluster ID: 583f2fa7e2ba41c498e17fb0c2f98850
I20260924 20:13:37.756053 3719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:37.762519 3719 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:37.763824 3719 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:37.774585 3719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Generated new TSK 0
I20260924 20:13:37.775475 3719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
I20260924 20:13:37.806921 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:37.923925 3731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:37.924199 3731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:37.924244 3731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:37.928377 3731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:37.928450 3731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:37.928512 3731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:37.933987 3731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:37.935051 3731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:37.936043 3731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:37.945916 3737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:37.945972 3739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:37.946177 3731 server_base.cc:1034] running on GCE node
W20260924 20:13:37.945912 3736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:37.946705 3731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:37.947528 3731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:37.948709 3731 hybrid_clock.cc:648] HybridClock initialized: now 1790280817948693 us; error 28 us; skew 500 ppm
Sep 24 20:13:37 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:37.951700 3731 init.cc:377] Logged in from keytab as oryx/127.29.83.129@KRBTEST.COM (short username oryx)
I20260924 20:13:37.952546 3731 webserver.cc:460] Webserver started at http://127.29.83.129:36919/ using document root <none> and password file <none>
I20260924 20:13:37.953058 3731 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:37.953114 3731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:37.953292 3731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:37.955317 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "832bdff82c8a4effa0aa38001b9a33df"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "181045b7116a304ed154a63aaeace778"
server_key_iv: "80e7ce2f842dcb84a4be52de1ad7ae16"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.955811 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "832bdff82c8a4effa0aa38001b9a33df"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "181045b7116a304ed154a63aaeace778"
server_key_iv: "80e7ce2f842dcb84a4be52de1ad7ae16"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.959508 3731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:13:37.962052 3746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:37.963442 3731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:37.963546 3731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "832bdff82c8a4effa0aa38001b9a33df"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "181045b7116a304ed154a63aaeace778"
server_key_iv: "80e7ce2f842dcb84a4be52de1ad7ae16"
server_key_version: "encryptionkey@0"
I20260924 20:13:37.963641 3731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:37.980501 3731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:37.984048 3731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:37.984217 3731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:37.984803 3731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:37.985585 3731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:37.985633 3731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:37.985673 3731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:37.985688 3731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:37.994588 3731 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:39239
I20260924 20:13:37.994649 3859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:39239 every 8 connection(s)
I20260924 20:13:37.995613 3731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:38.003669 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3731
I20260924 20:13:38.003808 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:38.004089 30030 external_mini_cluster.cc:1428] Setting key 323a6f9d3b401a64fb7e8c108486cd52
Sep 24 20:13:37 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:38.009269 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:37.997419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:55621 (local address 127.29.83.190:41729)
0924 20:13:37.997588 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:37.997593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:37.998443 (+ 850us) server_negotiation.cc:407] Connection header received
0924 20:13:37.999791 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:37.999793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:37.999840 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:37.999912 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.001077 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.001626 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.002829 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.003053 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.003128 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.005728 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.005731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.005733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.005797 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.007227 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.007737 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.007742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.007744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.007824 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.008172 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.008175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.008177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.008357 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.008468 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.008953 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.009077 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:13:38.010133 3862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:37.997849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:55621)
0924 20:13:37.998281 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:37.998325 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:37.999593 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.000079 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.000085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.000428 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.000941 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.000954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.001837 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.001843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.002715 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.002725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.003178 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.003180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.003324 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.003951 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.003970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.005531 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.007346 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.007352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.007367 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.007624 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.007923 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.007926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.007928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.008070 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.008464 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.008471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.008723 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.008915 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.009248 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
I20260924 20:13:38.011372 3860 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:38.011615 3860 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:38.012085 3860 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:38.013664 3669 ts_manager.cc:194] Registered new tserver with Master: 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129:39239)
I20260924 20:13:38.014922 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:55621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
I20260924 20:13:38.032961 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:38.145606 3865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:38.145905 3865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:38.145968 3865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:38.149976 3865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:38.150036 3865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:38.150100 3865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:38.155236 3865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:38.156212 3865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:38.157159 3865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:38.166527 3873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.166603 3871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.166553 3870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.166913 3865 server_base.cc:1034] running on GCE node
I20260924 20:13:38.167270 3865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:38.167884 3865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:38.169044 3865 hybrid_clock.cc:648] HybridClock initialized: now 1790280818169033 us; error 27 us; skew 500 ppm
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:38.171840 3865 init.cc:377] Logged in from keytab as oryx/127.29.83.130@KRBTEST.COM (short username oryx)
I20260924 20:13:38.172959 3865 webserver.cc:460] Webserver started at http://127.29.83.130:43311/ using document root <none> and password file <none>
I20260924 20:13:38.173628 3865 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.173691 3865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.173941 3865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:38.175959 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "701f1309378a415caab7efb0c3dd9d81"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "cdf99a92200857cb56273ea7a776f872"
server_key_iv: "1dce149ec7867481e932b6ed0183fd82"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.176529 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "701f1309378a415caab7efb0c3dd9d81"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "cdf99a92200857cb56273ea7a776f872"
server_key_iv: "1dce149ec7867481e932b6ed0183fd82"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.180238 3865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:13:38.182621 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.183789 3865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:38.183903 3865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "701f1309378a415caab7efb0c3dd9d81"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "cdf99a92200857cb56273ea7a776f872"
server_key_iv: "1dce149ec7867481e932b6ed0183fd82"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.184005 3865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.208339 3865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:38.211887 3865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:38.212102 3865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:38.212745 3865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:38.213598 3865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:38.213647 3865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.213686 3865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:38.213742 3865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.223127 3865 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:34285
I20260924 20:13:38.223150 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:34285 every 8 connection(s)
I20260924 20:13:38.224054 3865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:38.229938 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3865
I20260924 20:13:38.230083 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:38.230371 30030 external_mini_cluster.cc:1428] Setting key e7d3b0b80a227de17c0d148d8d5cd258
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:38.239873 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.226051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:48717 (local address 127.29.83.190:41729)
0924 20:13:38.226229 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.226235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.227178 (+ 943us) server_negotiation.cc:407] Connection header received
0924 20:13:38.228629 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.228632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.228687 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.228764 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.230049 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.230812 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.232325 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.232711 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.232837 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.235531 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.235537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.235543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.235578 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.237378 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.237999 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.238005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.238006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.238110 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.238464 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.238471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.238476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.238733 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.238835 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.239619 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.239747 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:13:38.241216 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.226532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:48717)
0924 20:13:38.227006 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.227050 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.228350 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.228929 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.228936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.229306 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.229893 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.229909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.230958 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.230963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.232180 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.232189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.232861 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.232863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.233012 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.233650 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.233678 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.235361 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.237504 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.237509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.237523 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.237881 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.238225 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.238227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.238228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.238359 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.238904 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.238914 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.239296 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.239540 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.239947 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:38.242717 3994 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:38.242976 3994 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:38.243484 3994 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.131@KRBTEST.COM" created.
I20260924 20:13:38.244642 3669 ts_manager.cc:194] Registered new tserver with Master: 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:38.245353 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:48717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
I20260924 20:13:38.260543 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:38.371842 3999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:38.372133 3999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:38.372179 3999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:38.376432 3999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:38.376502 3999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:38.376602 3999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:38.382128 3999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:38.383246 3999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:38.384344 3999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:38.394639 3999 server_base.cc:1034] running on GCE node
W20260924 20:13:38.394572 4004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.394587 4005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.394544 4007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.395287 3999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:38.395990 3999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:38.397258 3999 hybrid_clock.cc:648] HybridClock initialized: now 1790280818397242 us; error 40 us; skew 500 ppm
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:38.400247 3999 init.cc:377] Logged in from keytab as oryx/127.29.83.131@KRBTEST.COM (short username oryx)
I20260924 20:13:38.401240 3999 webserver.cc:460] Webserver started at http://127.29.83.131:40681/ using document root <none> and password file <none>
I20260924 20:13:38.401917 3999 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.401988 3999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.402220 3999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:38.404335 3999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "9331c8ccabea425db52bd9da870468bb"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "0c61b1e8aeef700644ecb64a3d40a990"
server_key_iv: "8d34751d95eb2db23b4f4ba7044d8698"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.404911 3999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "9331c8ccabea425db52bd9da870468bb"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "0c61b1e8aeef700644ecb64a3d40a990"
server_key_iv: "8d34751d95eb2db23b4f4ba7044d8698"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.408864 3999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 20:13:38.411585 4014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.412916 3999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:38.413091 3999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "9331c8ccabea425db52bd9da870468bb"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "0c61b1e8aeef700644ecb64a3d40a990"
server_key_iv: "8d34751d95eb2db23b4f4ba7044d8698"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.413220 3999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.441426 3999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:38.444810 3999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:38.445029 3999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:38.445789 3999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:38.446731 3999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:38.446797 3999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.446868 3999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:38.446900 3999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.456924 3999 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:45533
I20260924 20:13:38.456941 4127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:45533 every 8 connection(s)
I20260924 20:13:38.457975 3999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:38.467453 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 3999
I20260924 20:13:38.467557 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:38.467849 30030 external_mini_cluster.cc:1428] Setting key 264b9bc284c55a2c6ec69c60176a83ba
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:38.472661 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.459883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:52317 (local address 127.29.83.190:41729)
0924 20:13:38.460045 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.460050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.460964 (+ 914us) server_negotiation.cc:407] Connection header received
0924 20:13:38.462594 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.462596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.462647 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.462725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.464011 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.464536 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.465892 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.466094 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.466185 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.468896 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.468900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.468903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.468931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.470539 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.471120 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.471123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.471125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.471216 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.471595 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.471597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.471599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.471783 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.471862 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.472429 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.472538 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:13:38.473541 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.460342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:52317)
0924 20:13:38.460792 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.460838 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.462352 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.462971 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.462979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.463361 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.463868 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.463883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.464651 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.464654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.465735 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.465745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.466264 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.466267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.466413 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.466965 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.466981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.468724 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.470678 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.470683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.470696 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.471001 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.471324 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.471327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.471328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.471485 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.471875 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.471880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.472176 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.472352 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.472615 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260924 20:13:38.474787 4128 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:38.475096 4128 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:38.475574 4128 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
I20260924 20:13:38.476701 3669 ts_manager.cc:194] Registered new tserver with Master: 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533)
I20260924 20:13:38.477309 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:52317
I20260924 20:13:38.481971 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.83.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.189.keytab.
Entry for principal oryx/127.29.83.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.189.keytab.
I20260924 20:13:38.509449 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:39599
--webserver_interface=127.29.83.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.189.keytab
--principal=oryx/127.29.83.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41729,127.29.83.189:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:38.623943 4133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:38.624289 4133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:38.624374 4133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:38.628607 4133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:38.628767 4133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:38.628810 4133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:38.628872 4133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:38.628911 4133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:38.634482 4133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41729,127.29.83.189:39599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.189.keytab
--principal=oryx/127.29.83.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.83.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
W20260924 20:13:38.634759 4133 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.83.190:41729,127.29.83.189:39599'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 20:13:38.635648 4133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:38.636677 4133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:38.645181 4138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.645313 4141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.645361 4133 server_base.cc:1034] running on GCE node
W20260924 20:13:38.645239 4139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.646067 4133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:38.647186 4133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:38.648391 4133 hybrid_clock.cc:648] HybridClock initialized: now 1790280818648377 us; error 24 us; skew 500 ppm
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:38.651262 4133 init.cc:377] Logged in from keytab as oryx/127.29.83.189@KRBTEST.COM (short username oryx)
I20260924 20:13:38.652626 4133 webserver.cc:460] Webserver started at http://127.29.83.189:34443/ using document root <none> and password file <none>
I20260924 20:13:38.653206 4133 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.653301 4133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.653554 4133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:38.655850 4133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/instance:
uuid: "f7872da034d94c15a770e672e8885085"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "136aaf2cdcfe036d743b1ac4e006e0c6"
server_key_iv: "74fe5264968da4a669960d7b5e4b4939"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.656430 4133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal/instance:
uuid: "f7872da034d94c15a770e672e8885085"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "136aaf2cdcfe036d743b1ac4e006e0c6"
server_key_iv: "74fe5264968da4a669960d7b5e4b4939"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.660250 4133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:38.662838 4148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.664189 4133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:38.664357 4133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "f7872da034d94c15a770e672e8885085"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "136aaf2cdcfe036d743b1ac4e006e0c6"
server_key_iv: "74fe5264968da4a669960d7b5e4b4939"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.664505 4133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.679474 4133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:38.682760 4133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:38.682973 4133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:38.690321 4133 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.189:39599
I20260924 20:13:38.690603 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.189:39599 every 8 connection(s)
I20260924 20:13:38.691382 4133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
I20260924 20:13:38.694386 4201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:38.696195 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 4133
I20260924 20:13:38.699307 4201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.189@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:38.710423 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.700997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.29.83.190:41729)
0924 20:13:38.701137 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.701143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.701456 (+ 313us) server_negotiation.cc:407] Connection header received
0924 20:13:38.701609 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.701612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.701663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.701756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.702495 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.702998 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.704010 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.704350 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.704473 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.706779 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.706783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.706784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.706807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.708397 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.709021 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.709026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.709029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.709129 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.709412 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.709414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.709415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.709615 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.709746 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.710127 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.710239 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:13:38.711396 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.700944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32836)
0924 20:13:38.701334 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.701370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.701510 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.701831 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.701837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.702258 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.702375 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.702381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.703119 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.703122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.703864 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.703877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.704506 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.704509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.704587 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.705092 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.705107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.706624 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.708503 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.708509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.708517 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.708813 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.709227 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.709241 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.709243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.709319 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.709718 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.709725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.709882 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.710062 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.710406 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":76,"thread_start_us":146,"threads_started":1}
I20260924 20:13:38.711396 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.697613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.29.83.189:39599)
0924 20:13:38.698124 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.698140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.698177 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:13:38.698828 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.698845 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.699176 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.699537 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.700345 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.701900 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.702752 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.703223 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.703396 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.706160 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.706166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.706175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.706206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.708167 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.708640 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.708645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.708647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.708766 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.709144 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.709147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.709148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.709623 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.709882 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.710064 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.710358 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":175,"threads_started":1}
I20260924 20:13:38.713531 4201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.189@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
I20260924 20:13:38.721999 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.714074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42562)
0924 20:13:38.714223 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.714237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.714313 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.714516 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.714519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.714687 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.715122 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.715128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.715861 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.715864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.716378 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.716385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.716844 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.716847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.716982 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.717493 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.717506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.718769 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.720650 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.720653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.720654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.720851 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.721108 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.721110 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.721111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.721177 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.721551 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.721553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.721622 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.721715 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.721879 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
I20260924 20:13:38.722112 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.714171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.29.83.189:39599)
0924 20:13:38.714265 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.714271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.714285 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:38.714383 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.714387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.714420 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.714510 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.715249 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.715753 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.716487 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.716731 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.716835 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.718887 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.718890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.718892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.718912 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.720545 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.720940 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.720943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.720945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.721027 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.721265 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.721268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.721269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.721464 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.721560 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.721832 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.721943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":35}
I20260924 20:13:38.725481 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Bootstrap starting.
I20260924 20:13:38.728163 4201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:38.729009 4201 log.cc:824] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:38.731143 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: No bootstrap required, opened a new log
I20260924 20:13:38.734903 4201 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } }
I20260924 20:13:38.735111 4201 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:38.735145 4201 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7872da034d94c15a770e672e8885085, State: Initialized, Role: FOLLOWER
I20260924 20:13:38.735538 4201 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } }
I20260924 20:13:38.735975 4207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } } }
I20260924 20:13:38.736120 4207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:38.736733 4201 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:13:38.741479 4218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:38.741567 4218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:38.743494 4133 master_runner.cc:428] Detected that this master f7872da034d94c15a770e672e8885085 is joining an existing cluster
I20260924 20:13:38.743574 4133 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.83.189:39599
I20260924 20:13:38.754283 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.745654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.29.83.190:41729)
0924 20:13:38.745803 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.745808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.746212 (+ 404us) server_negotiation.cc:407] Connection header received
0924 20:13:38.746421 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.746425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.746478 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.746562 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.747215 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.747971 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.748840 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.749081 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.749197 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.750679 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.750682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.750685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.750710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.752350 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.752984 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.752988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.752990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.753080 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.753376 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.753379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.753381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.753612 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.753730 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.754021 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.754134 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:38.754611 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.745896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42578)
0924 20:13:38.746182 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.746195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.746331 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.746577 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.746580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.746885 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.747017 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.747023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.748032 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.748036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.749045 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.749057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.749658 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.749662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.749852 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.750285 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.750300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.750785 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.752883 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.752889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.752892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.753101 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.753408 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.753411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.753413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.753485 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.753915 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.753918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.753990 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.754323 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.754442 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":52,"thread_start_us":87,"threads_started":1}
I20260924 20:13:38.754300 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.745612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32844)
0924 20:13:38.746111 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.746126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.746247 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.746684 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.746688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.746974 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.747104 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.747111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.748092 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.748094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.748713 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.748721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.749340 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.749344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.749498 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.750063 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.750074 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.750539 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.752480 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.752484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.752486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.752815 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.753179 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.753184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.753188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.753272 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.753721 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.753724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.753840 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.753957 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.754104 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":65,"spinlock_wait_cycles":17792,"thread_start_us":100,"threads_started":1}
I20260924 20:13:38.755264 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.745734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.29.83.189:39599)
0924 20:13:38.745959 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.745962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.746292 (+ 330us) server_negotiation.cc:407] Connection header received
0924 20:13:38.746434 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.746438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.746473 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.746564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.747127 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.747908 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.749174 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.749537 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.749646 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.750914 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.750918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.750921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.750940 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.752780 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.753232 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.753235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.753237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.753314 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.753569 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.753572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.753573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.753791 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.753858 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.754134 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.754238 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":171}
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.190@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
I20260924 20:13:38.770025 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.759322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.29.83.189:39599)
0924 20:13:38.759442 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.759446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.759757 (+ 311us) server_negotiation.cc:407] Connection header received
0924 20:13:38.759939 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.759942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.759997 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.760078 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.760711 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.761462 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.762401 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.762741 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.762838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.765598 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.765601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.765604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.765629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.767355 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.767961 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.767965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.767967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.768054 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.768558 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.768564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.768568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.768809 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.768905 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.769657 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.769816 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:38.770594 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.759257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42588)
0924 20:13:38.759627 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.759679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.759811 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.760118 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.760122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.760478 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.760596 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.760602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.761574 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.761576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.762296 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.762305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.762858 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.762861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.762953 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.763531 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.763552 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.765465 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.767499 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.767511 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.767520 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.767842 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.768180 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.768184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.768282 (+ 98us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.768391 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.768962 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.768972 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.769120 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.769302 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.770375 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260924 20:13:38.772367 3669 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.83.189:39599: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7872da034d94c15a770e672e8885085"
member_type: NON_VOTER
last_known_addr {
host: "127.29.83.189"
port: 39599
}
attrs {
promote: true
}
}
dest_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4"
cas_config_opid_index: -1
I20260924 20:13:38.773142 3669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.29.83.188@KRBTEST.COM; defaulting to no policy
I20260924 20:13:38.775081 4229 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f7872da034d94c15a770e672e8885085 (127.29.83.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } }
I20260924 20:13:38.775893 4175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:38.777351 4229 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.83.189:39599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.188@KRBTEST.COM" created.
I20260924 20:13:38.777863 4228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } }
I20260924 20:13:38.777987 4228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:38.778136 4175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 20:13:38.778749 4133 master.cc:537] Master@127.29.83.189:39599 shutting down...
I20260924 20:13:38.778967 4228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:13:38.781838 3656 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 20:13:38.783025 3656 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 -> Peer f7872da034d94c15a770e672e8885085 (127.29.83.189:39599): Couldn't send request to peer f7872da034d94c15a770e672e8885085. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 20:13:38.783746 4133 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:38.784083 4133 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:38.784214 4133 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: stopping tablet replica
I20260924 20:13:38.789543 4133 master.cc:552] Master@127.29.83.189:39599 shutdown complete.
I20260924 20:13:38.789634 4133 master_runner.cc:305] Clearing existing system tablet
I20260924 20:13:38.789938 4133 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.789997 4133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.791311 4133 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:38.792165 4234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.792400 4133 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.792522 4133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:38.792555 4133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "f7872da034d94c15a770e672e8885085"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "136aaf2cdcfe036d743b1ac4e006e0c6"
server_key_iv: "74fe5264968da4a669960d7b5e4b4939"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.794667 4133 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.188.keytab.
Entry for principal oryx/127.29.83.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.188.keytab.
I20260924 20:13:38.797251 4133 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:13:38.797345 4133 log.cc:1196] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 20:13:38.797523 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:34929
--webserver_interface=127.29.83.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.188.keytab
--principal=oryx/127.29.83.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:13:38.797590 4133 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Deleting consensus metadata
I20260924 20:13:38.797780 4133 master_runner.cc:315] Copying system tablet from 127.29.83.190:41729
I20260924 20:13:38.798983 4133 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: tablet copy: Beginning tablet copy session from remote peer at address 127.29.83.190:41729
I20260924 20:13:38.808895 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.800000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.29.83.190:41729)
0924 20:13:38.800152 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:38.800156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:38.800172 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:13:38.800225 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:38.800231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:38.800291 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:38.800370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:38.800847 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.801372 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.802665 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.803097 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.803233 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.804559 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:38.804566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:38.804572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:38.804607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:38.806610 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.807214 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.807220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.807223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.807321 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:38.807692 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:38.807696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:38.807700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:38.807972 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:38.808070 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:38.808498 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 20:13:38.808648 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:13:38.809670 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:38.799589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32848)
0924 20:13:38.799869 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:38.799886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:38.799995 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:38.800411 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:38.800415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:38.800620 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:38.800733 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.800740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.801475 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.801478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.802520 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:38.802531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:38.803231 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:38.803235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:38.803326 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:38.803947 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:38.803962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:38.804393 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:38.806760 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.806766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.806769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.807079 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.807450 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:38.807455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:38.807457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:38.807553 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:38.808154 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:38.808160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:38.808281 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 20:13:38.809177 (+ 896us) client_negotiation.cc:240] Negotiation successful
0924 20:13:38.809386 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260924 20:13:38.810344 3689 tablet_copy_service.cc:140] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7872da034d94c15a770e672e8885085 ({username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:32848)
I20260924 20:13:38.810551 3689 tablet_copy_service.cc:161] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7872da034d94c15a770e672e8885085 at {username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:32848: session id = f7872da034d94c15a770e672e8885085-00000000000000000000000000000000
I20260924 20:13:38.812386 3689 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:13:38.814232 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:13:38.816668 4133 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: tablet copy: Starting download of 0 data blocks...
I20260924 20:13:38.816879 4133 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: tablet copy: Starting download of 1 WAL segments...
I20260924 20:13:38.818925 4133 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:13:38.820931 3689 tablet_copy_service.cc:342] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Request end of tablet copy session f7872da034d94c15a770e672e8885085-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:32848
I20260924 20:13:38.821085 3689 tablet_copy_service.cc:433] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: ending tablet copy session f7872da034d94c15a770e672e8885085-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7872da034d94c15a770e672e8885085
I20260924 20:13:38.828655 4133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:38.830447 4242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.830657 4245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.830837 4243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.834769 4133 server_base.cc:1034] running on GCE node
I20260924 20:13:38.834931 4133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:38.835222 4133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:38.836361 4133 hybrid_clock.cc:648] HybridClock initialized: now 1790280818836344 us; error 35 us; skew 500 ppm
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:38.837992 4133 init.cc:377] Logged in from keytab as oryx/127.29.83.189@KRBTEST.COM (short username oryx)
I20260924 20:13:38.838488 4133 webserver.cc:460] Webserver started at http://127.29.83.189:35535/ using document root <none> and password file <none>
I20260924 20:13:38.838713 4133 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.838757 4133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.840102 4133 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:38.841171 4252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.841413 4133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:38.841522 4133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "f7872da034d94c15a770e672e8885085"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "136aaf2cdcfe036d743b1ac4e006e0c6"
server_key_iv: "74fe5264968da4a669960d7b5e4b4939"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.841670 4133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.851716 4133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:38.854185 4133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:38.858728 4133 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.189:39599
I20260924 20:13:38.858963 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.189:39599 every 8 connection(s)
I20260924 20:13:38.859339 4133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
I20260924 20:13:38.861073 4305 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:13:38.862174 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Bootstrap starting.
I20260924 20:13:38.872746 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:13:38.873099 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Bootstrap complete.
I20260924 20:13:38.873742 4305 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } }
I20260924 20:13:38.873915 4305 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7872da034d94c15a770e672e8885085, State: Initialized, Role: LEARNER
I20260924 20:13:38.874029 4305 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } }
I20260924 20:13:38.874487 4305 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:38.874473 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } }
I20260924 20:13:38.874822 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:13:38.877861 4319 catalog_manager.cc:1259] Loaded cluster ID: 583f2fa7e2ba41c498e17fb0c2f98850
I20260924 20:13:38.877920 4319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: loading cluster ID for follower catalog manager: success
I20260924 20:13:38.879951 4319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: acquiring CA information for follower catalog manager: success
I20260924 20:13:38.881269 4319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 20:13:38.933619 4239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:38.933936 4239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:38.934026 4239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:38.938387 4239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:38.938457 4239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:38.938488 4239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:38.938545 4239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:38.938581 4239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:38.944522 4239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.188.keytab
--principal=oryx/127.29.83.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.83.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:38.945842 4239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:38.947031 4239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:38.956149 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:38.956147 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.956347 4239 server_base.cc:1034] running on GCE node
W20260924 20:13:38.956547 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:38.957194 4239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:38.958460 4239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:38.959667 4239 hybrid_clock.cc:648] HybridClock initialized: now 1790280818959645 us; error 35 us; skew 500 ppm
Sep 24 20:13:38 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:38.962749 4239 init.cc:377] Logged in from keytab as oryx/127.29.83.188@KRBTEST.COM (short username oryx)
I20260924 20:13:38.963696 4239 webserver.cc:460] Webserver started at http://127.29.83.188:42011/ using document root <none> and password file <none>
I20260924 20:13:38.964345 4239 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:38.964417 4239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:38.964596 4239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:38.966915 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/instance:
uuid: "2c16523db3f84ba1821c6cbf98b063c0"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "8f4d6d8d59ce8d668a5d775abceb6169"
server_key_iv: "c2cf914f927d79597f5df0898601e10e"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.967537 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal/instance:
uuid: "2c16523db3f84ba1821c6cbf98b063c0"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "8f4d6d8d59ce8d668a5d775abceb6169"
server_key_iv: "c2cf914f927d79597f5df0898601e10e"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.971498 4239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:38.974041 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:38.975658 4239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:38.975790 4239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "2c16523db3f84ba1821c6cbf98b063c0"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "8f4d6d8d59ce8d668a5d775abceb6169"
server_key_iv: "c2cf914f927d79597f5df0898601e10e"
server_key_version: "encryptionkey@0"
I20260924 20:13:38.975908 4239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:38.993228 4239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:38.997217 4239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:38.997419 4239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:39.005475 4239 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.188:34929
I20260924 20:13:39.005939 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.188:34929 every 8 connection(s)
I20260924 20:13:39.007081 4239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
I20260924 20:13:39.010052 4387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:39.013706 4387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:13:39.017225 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 4239
I20260924 20:13:39.017488 3860 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.188@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
I20260924 20:13:39.029889 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.015719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.29.83.190:41729)
0924 20:13:39.015865 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.015869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.016724 (+ 855us) server_negotiation.cc:407] Connection header received
0924 20:13:39.017901 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.017904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.017962 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.018068 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.019636 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.020300 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.021915 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.022176 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.022282 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.025598 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.025604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.025611 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.025649 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.027422 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.027966 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.027969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.027970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.028048 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.028449 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.028453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.028455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.028715 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.028806 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.029533 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.029670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:39.030956 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.016008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32860)
0924 20:13:39.016550 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.016593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.017610 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.018238 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.018247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.018728 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.019425 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.019445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.020442 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.020445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.021803 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.021814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.022317 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.022320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.022854 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.023532 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.023553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.025411 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.027530 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.027537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.027550 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.027864 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.028144 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.028149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.028154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.028332 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.028821 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.028828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.029218 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.029455 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.029832 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":230,"spinlock_wait_cycles":6784,"thread_start_us":152,"threads_started":1}
I20260924 20:13:39.030956 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.018492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.29.83.188:34929)
0924 20:13:39.018817 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.018829 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.018850 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:13:39.018889 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.018894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.019067 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.019193 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.020035 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.021095 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.022156 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.022633 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.022751 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.026140 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.026148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.026163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.026191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.028193 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.028734 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.028738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.028739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.028836 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.029084 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.029087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.029089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.029366 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.029544 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.029819 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.029939 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"spinlock_wait_cycles":384,"thread_start_us":155,"threads_started":1}
I20260924 20:13:39.032501 4387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.188@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
I20260924 20:13:39.041632 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.033085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42598)
0924 20:13:39.033225 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.033244 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.033334 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.033897 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.033900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.034093 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.034200 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.034206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.035267 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.035279 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.035873 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.035881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.036462 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.036467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.036609 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.037085 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.037099 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.038474 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.040369 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.040372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.040373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.040658 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.040901 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.040903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.040904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.040974 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.041256 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.041258 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.041332 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.041409 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.041509 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:13:39.041823 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.033173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.29.83.189:39599)
0924 20:13:39.033604 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.033608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.033621 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:13:39.033661 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.033663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.033706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.033868 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.034338 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.035122 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.036077 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.036337 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.036464 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.038616 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.038619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.038622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.038650 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.040273 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.040742 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.040744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.040746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.040824 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.041029 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.041031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.041032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.041188 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.041255 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.041476 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.041574 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":96,"threads_started":1}
I20260924 20:13:39.042547 4387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280818, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.188@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
I20260924 20:13:39.050026 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.043101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59550)
0924 20:13:39.043207 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.043220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.043290 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.043466 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.043467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.043587 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.043655 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.043660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.044328 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.044330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.044820 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.044826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.045311 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.045314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.045370 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.045715 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.045725 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.046990 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.048572 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.048575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.048576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.048781 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.049128 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.049131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.049132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.049228 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.049565 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.049567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.049643 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.049732 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.049904 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 20:13:39.050168 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.043152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.29.83.188:34929)
0924 20:13:39.043268 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.043271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.043283 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:39.043363 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.043365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.043396 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.043459 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.043738 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.044233 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.044968 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.045216 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.045297 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.047106 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.047109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.047110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.047209 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.048369 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.048907 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.048910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.048911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.048982 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.049324 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.049326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.049327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.049483 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.049555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.049927 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.050018 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:13:39.053718 4387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Bootstrap starting.
I20260924 20:13:39.056409 4387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:39.057276 4387 log.cc:824] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:39.059454 4387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: No bootstrap required, opened a new log
I20260924 20:13:39.063108 4387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } }
I20260924 20:13:39.063434 4387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:39.063537 4387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c16523db3f84ba1821c6cbf98b063c0, State: Initialized, Role: FOLLOWER
I20260924 20:13:39.064100 4387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } }
I20260924 20:13:39.064639 4393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } } }
I20260924 20:13:39.064766 4393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:39.065429 4387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:13:39.070603 4404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:13:39.070710 4404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:13:39.074193 4239 master_runner.cc:428] Detected that this master 2c16523db3f84ba1821c6cbf98b063c0 is joining an existing cluster
I20260924 20:13:39.074281 4239 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.83.188:34929
I20260924 20:13:39.084892 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3731
I20260924 20:13:39.092229 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:39239
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:41729
--webserver_port=36919
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:13:39.108134 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.076690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42610)
0924 20:13:39.077331 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.077343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.077415 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.078104 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.078107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.078375 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.078569 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.078577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079667 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.079670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.081233 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.081244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081969 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.082099 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.102489 (+ 20390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.102510 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.103111 (+ 601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.105711 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.105715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.105717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.106008 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.107044 (+ 1036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.107047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.107049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.107131 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.107568 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.107571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.107668 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.107776 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.107929 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":166,"thread_start_us":90,"threads_started":1}
I20260924 20:13:39.108593 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.077707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.29.83.190:41729)
0924 20:13:39.077874 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.077878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.077892 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:39.077941 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.077944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.077990 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.078067 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.078781 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079602 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081088 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081463 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081537 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.104453 (+ 22916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.104458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.104461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.104496 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.106652 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.107145 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.107149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.107150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.107236 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.107530 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.107533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.107535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.107761 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.107828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.108158 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.108438 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 20:13:39.108666 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.077569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.29.83.189:39599)
0924 20:13:39.077891 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.077895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.077907 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:39.077943 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.077945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.077981 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.078084 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.078711 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079531 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081371 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081746 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081840 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.103364 (+ 21524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.103369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.103372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.103405 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.105250 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.106852 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.106856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.106858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.106945 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.107223 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.107226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.107228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.107444 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.107526 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.108378 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.108510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260924 20:13:39.109498 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.076681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32862)
0924 20:13:39.077006 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.077023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.077127 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.078188 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.078191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.078452 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.078633 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.078641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079780 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.079783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.080953 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.080963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081732 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.081866 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.082391 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.082408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.104033 (+ 21625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.106798 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.106801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.106803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.107032 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.107343 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.107346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.107348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.107435 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.107897 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.107900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.107988 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.109146 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.109320 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":72,"thread_start_us":117,"threads_started":1}
I20260924 20:13:39.111809 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.077616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.29.83.188:34929)
0924 20:13:39.077708 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.077713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.078414 (+ 701us) server_negotiation.cc:407] Connection header received
0924 20:13:39.078548 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.078551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.078584 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.078629 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.079182 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079873 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081396 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081704 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.104319 (+ 22516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.104323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.104326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.104352 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.106747 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.110728 (+ 3981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.110731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.110733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.110814 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.111056 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.111058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.111060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.111237 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.111309 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.111547 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.111652 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260924 20:13:39.112373 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.077572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59562)
0924 20:13:39.078336 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.078351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.078458 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.078741 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.078744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.078962 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.079059 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.079065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.079999 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.080003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.081299 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.081308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.081961 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.081964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.082395 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.082851 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.082866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.103522 (+ 20656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.109634 (+ 6112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.109637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.109639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.110136 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.110909 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.110912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.110913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.110982 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.111347 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.111349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.111416 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.111963 (+ 547us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.112083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":456,"thread_start_us":108,"threads_started":1}
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280817, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.190@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
I20260924 20:13:39.121685 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.113746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59576)
0924 20:13:39.113923 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.113938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.114028 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.114325 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.114328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.114501 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.114583 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.114588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.115287 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.115292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.115836 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.115844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.116304 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.116307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.116404 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.116966 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.116979 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.118386 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.120165 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.120169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.120171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.120379 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.120658 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.120660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.120663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.120757 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.121135 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.121138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.121248 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.121356 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.121516 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 20:13:39.121685 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.113807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.29.83.188:34929)
0924 20:13:39.113934 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.113939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.113981 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:13:39.114152 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.114155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.114209 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.114309 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.114692 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.115177 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.115953 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.116183 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.116295 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.118529 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.118540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.118543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.118564 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.120052 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.120476 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.120481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.120483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.120558 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.120853 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.120856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.120858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.121038 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.121144 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.121406 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.121513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:13:39.123037 3669 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.83.188:34929: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0"
member_type: NON_VOTER
last_known_addr {
host: "127.29.83.188"
port: 34929
}
attrs {
promote: true
}
}
dest_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4"
cas_config_opid_index: 5
I20260924 20:13:39.123618 3669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } }
I20260924 20:13:39.124692 4228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.125926 4228 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.83.188:34929
I20260924 20:13:39.126101 4228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.126231 4228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.127372 4361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:39.128491 4361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:13:39.129113 4229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:39.129628 4239 master.cc:537] Master@127.29.83.188:34929 shutting down...
W20260924 20:13:39.131350 3656 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 -> Peer 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929): Couldn't send request to peer 2c16523db3f84ba1821c6cbf98b063c0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 20:13:39.133538 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.126768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42618)
0924 20:13:39.126878 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.126893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.126966 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.130004 (+ 3038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.130011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.130035 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.130130 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.130136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.131156 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.131159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.132216 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.132226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.133065 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.133069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.133151 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.133208 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.133284 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.133377 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 20:13:39.133909 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.128945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.29.83.189:39599)
0924 20:13:39.129078 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.129082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.129097 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:39.129150 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.129152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.129227 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.129341 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.130282 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.131022 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.132341 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.132944 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.133066 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.133215 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.133674 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.133752 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:13:39.134513 4279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 LEARNER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:13:39.135078 4229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 20:13:39.135215 4239 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:39.135480 4239 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:39.135593 4239 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: stopping tablet replica
I20260924 20:13:39.137111 4279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.138731 4415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.138845 4415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:13:39.138744 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: NON_VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: true } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.138981 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:13:39.140641 4228 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: attempting to promote NON_VOTER f7872da034d94c15a770e672e8885085 to VOTER
I20260924 20:13:39.141155 4228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } }
I20260924 20:13:39.142679 4279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 LEARNER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 20:13:39.142841 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.142512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59592)
0924 20:13:39.142677 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.142759 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:34929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 20:13:39.143170 4228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 20:13:39.143790 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.143341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59596)
0924 20:13:39.143670 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.143730 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:34929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
W20260924 20:13:39.143987 3656 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 -> Peer 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929): Couldn't send request to peer 2c16523db3f84ba1821c6cbf98b063c0. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.188:34929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:13:39.144779 4228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7872da034d94c15a770e672e8885085 (127.29.83.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.146407 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.146503 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.146219 4228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.146744 4228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.147032 4279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7872da034d94c15a770e672e8885085 (127.29.83.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.148279 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.148388 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:39.153299 4239 master.cc:552] Master@127.29.83.188:34929 shutdown complete.
I20260924 20:13:39.153378 4239 master_runner.cc:305] Clearing existing system tablet
I20260924 20:13:39.153599 4239 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:39.153669 4239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:39.154762 4239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:39.155596 4418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.155825 4239 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:39.155953 4239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:39.155995 4239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "2c16523db3f84ba1821c6cbf98b063c0"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "8f4d6d8d59ce8d668a5d775abceb6169"
server_key_iv: "c2cf914f927d79597f5df0898601e10e"
server_key_version: "encryptionkey@0"
I20260924 20:13:39.157528 4239 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:39.159341 4239 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:13:39.159451 4239 log.cc:1196] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 20:13:39.159677 4239 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Deleting consensus metadata
I20260924 20:13:39.159866 4239 master_runner.cc:315] Copying system tablet from 127.29.83.190:41729
I20260924 20:13:39.160964 4239 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: tablet copy: Beginning tablet copy session from remote peer at address 127.29.83.190:41729
I20260924 20:13:39.168968 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.161584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.29.83.190:41729)
0924 20:13:39.161738 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.161744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.161950 (+ 206us) server_negotiation.cc:407] Connection header received
0924 20:13:39.162076 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.162079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.162125 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.162226 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.162673 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.163207 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.164034 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.164377 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.164492 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.165729 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.165733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.165736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.165798 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.167363 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.167760 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.167764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.167766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.167850 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.168132 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.168135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.168137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.168319 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.168420 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.168684 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.168804 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:13:39.168978 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.161518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:32868)
0924 20:13:39.161856 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.161869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.161995 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.162253 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.162256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.162454 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.162554 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.162559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.163317 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.163322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.163912 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.163919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.164507 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.164511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.164611 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.165154 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.165166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.165576 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.167453 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.167458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.167460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.167659 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.167947 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.167950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.167952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.168034 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.168420 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.168424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.168523 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.168628 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.168771 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":90,"threads_started":1}
I20260924 20:13:39.169725 3689 tablet_copy_service.cc:140] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c16523db3f84ba1821c6cbf98b063c0 ({username='oryx', principal='oryx/127.29.83.188@KRBTEST.COM'} at 127.0.0.1:32868)
I20260924 20:13:39.169873 3689 tablet_copy_service.cc:161] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c16523db3f84ba1821c6cbf98b063c0 at {username='oryx', principal='oryx/127.29.83.188@KRBTEST.COM'} at 127.0.0.1:32868: session id = 2c16523db3f84ba1821c6cbf98b063c0-00000000000000000000000000000000
I20260924 20:13:39.170802 3689 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:13:39.171936 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:13:39.173843 4239 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: tablet copy: Starting download of 0 data blocks...
I20260924 20:13:39.174070 4239 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: tablet copy: Starting download of 1 WAL segments...
I20260924 20:13:39.175622 4239 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:13:39.177280 3689 tablet_copy_service.cc:342] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Request end of tablet copy session 2c16523db3f84ba1821c6cbf98b063c0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.83.188@KRBTEST.COM'} at 127.0.0.1:32868
I20260924 20:13:39.177386 3689 tablet_copy_service.cc:433] P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: ending tablet copy session 2c16523db3f84ba1821c6cbf98b063c0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c16523db3f84ba1821c6cbf98b063c0
I20260924 20:13:39.184144 4239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:39.185963 4425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.186035 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:39.186358 4239 server_base.cc:1034] running on GCE node
W20260924 20:13:39.186411 4426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:39.186992 4239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:39.187369 4239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:39.188508 4239 hybrid_clock.cc:648] HybridClock initialized: now 1790280819188499 us; error 27 us; skew 500 ppm
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:39.190050 4239 init.cc:377] Logged in from keytab as oryx/127.29.83.188@KRBTEST.COM (short username oryx)
I20260924 20:13:39.190575 4239 webserver.cc:460] Webserver started at http://127.29.83.188:38235/ using document root <none> and password file <none>
I20260924 20:13:39.190770 4239 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:39.190826 4239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:39.192056 4239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:39.192996 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.193239 4239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:39.193332 4239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "2c16523db3f84ba1821c6cbf98b063c0"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "8f4d6d8d59ce8d668a5d775abceb6169"
server_key_iv: "c2cf914f927d79597f5df0898601e10e"
server_key_version: "encryptionkey@0"
I20260924 20:13:39.193435 4239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:39.210801 4239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:39.212901 4239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:39.218039 4239 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.188:34929
I20260924 20:13:39.218609 4239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
I20260924 20:13:39.219528 4487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.188:34929 every 8 connection(s)
I20260924 20:13:39.220490 4488 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:13:39.221351 4488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Bootstrap starting.
I20260924 20:13:39.231361 4488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:13:39.231720 4488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: Bootstrap complete.
I20260924 20:13:39.232280 4488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } }
I20260924 20:13:39.232416 4488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c16523db3f84ba1821c6cbf98b063c0, State: Initialized, Role: LEARNER
I20260924 20:13:39.232529 4488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } }
I20260924 20:13:39.232869 4491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.233027 4488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:39.233036 4491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:13:39.236404 4502 catalog_manager.cc:1259] Loaded cluster ID: 583f2fa7e2ba41c498e17fb0c2f98850
I20260924 20:13:39.236482 4502 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: loading cluster ID for follower catalog manager: success
I20260924 20:13:39.238111 4502 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: acquiring CA information for follower catalog manager: success
I20260924 20:13:39.239319 4502 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:13:39.247939 3994 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
W20260924 20:13:39.255326 4413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:39.255682 4413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:39.255745 4413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:39.262028 4413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:39.262110 4413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:39.262212 4413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:39.270251 4413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:39239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=36919
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:39.272007 4413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:39.273253 4413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:39.284227 4507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.284250 4510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.284240 4508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:39.284402 4413 server_base.cc:1034] running on GCE node
I20260924 20:13:39.284894 4413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:39.285537 4413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:39.286687 4413 hybrid_clock.cc:648] HybridClock initialized: now 1790280819286650 us; error 50 us; skew 500 ppm
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:39.289454 4413 init.cc:377] Logged in from keytab as oryx/127.29.83.129@KRBTEST.COM (short username oryx)
I20260924 20:13:39.290445 4413 webserver.cc:460] Webserver started at http://127.29.83.129:36919/ using document root <none> and password file <none>
I20260924 20:13:39.291009 4413 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:39.291085 4413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:39.295282 4413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:13:39.297566 4517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.298919 4413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:39.299062 4413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "832bdff82c8a4effa0aa38001b9a33df"
format_stamp: "Formatted at 2026-09-24 20:13:37 on dist-test-slave-52fr"
server_key: "181045b7116a304ed154a63aaeace778"
server_key_iv: "80e7ce2f842dcb84a4be52de1ad7ae16"
server_key_version: "encryptionkey@0"
I20260924 20:13:39.299646 4413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:39.316478 4413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:39.319689 4413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:39.319908 4413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:39.320508 4413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:39.321362 4413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:39.321431 4413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.321501 4413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:39.321537 4413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.331051 4413 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:39239
I20260924 20:13:39.331070 4630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:39239 every 8 connection(s)
I20260924 20:13:39.332098 4413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:39.334198 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 4413
I20260924 20:13:39.334446 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3865
I20260924 20:13:39.344118 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:34285
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:41729
--webserver_port=43311
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
I20260924 20:13:39.355660 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.335292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:57157 (local address 127.29.83.189:39599)
0924 20:13:39.335513 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.335519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.338363 (+ 2844us) server_negotiation.cc:407] Connection header received
0924 20:13:39.339542 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.339546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.339632 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.339719 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.342025 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.342840 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.344888 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.345235 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345302 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.350125 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.350129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.350132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.350175 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.352628 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353503 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.353511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.353515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.353610 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.354104 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.354107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.354109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.354555 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.354676 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.355354 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.355491 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 20:13:39.355683 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.337716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:40123 (local address 127.29.83.188:34929)
0924 20:13:39.337995 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.338001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.341679 (+ 3678us) server_negotiation.cc:407] Connection header received
0924 20:13:39.341885 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.341891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.341956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.342019 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.342810 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.343867 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345155 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.345597 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345720 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.350088 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.350092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.350095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.350175 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.352709 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353555 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.353561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.353565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.353661 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.354102 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.354105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.354107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.354342 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.354402 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.355356 (+ 954us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.355492 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260924 20:13:39.355759 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.336017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:38537 (local address 127.29.83.190:41729)
0924 20:13:39.336240 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.336247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.337839 (+ 1592us) server_negotiation.cc:407] Connection header received
0924 20:13:39.339475 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.339482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.339561 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.339621 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.342027 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.342847 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.344818 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.345154 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345230 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.349651 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.349658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.349662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.349702 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.352929 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353639 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.353645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.353648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.353791 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.354149 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.354155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.354159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.354427 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.354521 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.355369 (+ 848us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.355516 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 20:13:39.357494 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.334693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:38537)
0924 20:13:39.337628 (+ 2935us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.337706 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.339201 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.340182 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.340195 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.340852 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.341881 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.341886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.343029 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.343033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.344655 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.344666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.345469 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.345803 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.346682 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.346743 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.349425 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.353086 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.353094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.353500 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.353924 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.353934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.354034 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.354601 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.354611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.355052 (+ 441us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.356570 (+ 1518us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.356768 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2661,"mutex_wait_us":629,"spinlock_wait_cycles":82432,"thread_start_us":152,"threads_started":1}
I20260924 20:13:39.357491 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.334693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.29.83.129:57157)
0924 20:13:39.338261 (+ 3568us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.338279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.339234 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.340228 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.340234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.340914 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.341836 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.341862 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.343029 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.343033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.344763 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.344775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.345468 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.345472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.345802 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.346714 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.346729 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.349962 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.352780 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.352788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.352809 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.353372 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.353727 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.353738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.353991 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.354658 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.354661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.355050 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.355872 (+ 822us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.356305 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3453,"mutex_wait_us":1693,"spinlock_wait_cycles":72192,"thread_start_us":216,"threads_started":1}
I20260924 20:13:39.357491 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.337343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.29.83.129:40123)
0924 20:13:39.341574 (+ 4231us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.341591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.341749 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.342232 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.342237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.342455 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.342667 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.342678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.343999 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.344005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.345016 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.345030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.346818 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.346823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.346988 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.347763 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.347784 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.349916 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.353005 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.353010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.353453 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.353794 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.353798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.353803 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.353991 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.354744 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.354746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.355050 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.355833 (+ 783us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.356360 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4139,"mutex_wait_us":210,"spinlock_wait_cycles":13824,"thread_start_us":102,"threads_started":1}
I20260924 20:13:39.359002 4633 heartbeater.cc:347] Connected to a master server at 127.29.83.189:39599
I20260924 20:13:39.359102 4631 heartbeater.cc:347] Connected to a master server at 127.29.83.188:34929
I20260924 20:13:39.359217 4632 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:39.359277 4633 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.359277 4631 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.359297 4632 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.359885 4633 heartbeater.cc:511] Master 127.29.83.189:39599 requested a full tablet report, sending...
I20260924 20:13:39.360014 4632 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
I20260924 20:13:39.359886 4631 heartbeater.cc:511] Master 127.29.83.188:34929 requested a full tablet report, sending...
I20260924 20:13:39.361238 3669 ts_manager.cc:194] Re-registered known tserver with Master: 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129:39239)
I20260924 20:13:39.361644 4269 ts_manager.cc:194] Registered new tserver with Master: 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129:39239)
I20260924 20:13:39.362051 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:38537
I20260924 20:13:39.362928 4452 ts_manager.cc:194] Registered new tserver with Master: 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129:39239)
W20260924 20:13:39.464061 4639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:39.464308 4639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:39.464351 4639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:39.468417 4639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:39.468480 4639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:39.468571 4639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:39.473899 4639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:34285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=43311
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:39.475088 4639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:39.476150 4639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:39.479555 4128 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
W20260924 20:13:39.486553 4644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.486452 4645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.486459 4647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:39.486799 4639 server_base.cc:1034] running on GCE node
I20260924 20:13:39.487291 4639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:39.488010 4639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:39.489217 4639 hybrid_clock.cc:648] HybridClock initialized: now 1790280819489159 us; error 71 us; skew 500 ppm
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:39.492242 4639 init.cc:377] Logged in from keytab as oryx/127.29.83.130@KRBTEST.COM (short username oryx)
I20260924 20:13:39.493232 4639 webserver.cc:460] Webserver started at http://127.29.83.130:43311/ using document root <none> and password file <none>
I20260924 20:13:39.493918 4639 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:39.494004 4639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:39.498438 4639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:13:39.500980 4654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.502808 4639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:39.502983 4639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "701f1309378a415caab7efb0c3dd9d81"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "cdf99a92200857cb56273ea7a776f872"
server_key_iv: "1dce149ec7867481e932b6ed0183fd82"
server_key_version: "encryptionkey@0"
I20260924 20:13:39.503587 4639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:39.526974 4639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:39.530773 4639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:39.531018 4639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:39.531685 4639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:39.532675 4639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:39.532745 4639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.532811 4639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:39.532863 4639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.542539 4639 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:34285
I20260924 20:13:39.542774 4767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:34285 every 8 connection(s)
I20260924 20:13:39.543676 4639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:39.552565 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 4639
I20260924 20:13:39.552780 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3999
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
I20260924 20:13:39.562060 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:45533
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:41729
--webserver_port=40681
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:13:39.563475 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.545646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:43111 (local address 127.29.83.189:39599)
0924 20:13:39.545804 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.545808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.548396 (+ 2588us) server_negotiation.cc:407] Connection header received
0924 20:13:39.549144 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.549147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.549208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.549289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.550853 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551675 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.553857 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554196 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554275 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.559513 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.559518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.559522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.559555 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.561256 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561818 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.561822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.561823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.561901 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.562212 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.562216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.562218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.562451 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.562535 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.563173 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.563307 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:39.563475 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.545635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:48783 (local address 127.29.83.190:41729)
0924 20:13:39.545798 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.545803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.547378 (+ 1575us) server_negotiation.cc:407] Connection header received
0924 20:13:39.548632 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.548637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.548810 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.548914 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.550868 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551737 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.553853 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554289 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554384 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.558329 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.558337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.558343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.558383 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.560770 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561456 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.561462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.561464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.561550 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.562031 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.562034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.562035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.562287 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.562366 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.563188 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.563307 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:13:39.563568 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.548548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:54553 (local address 127.29.83.188:34929)
0924 20:13:39.548672 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.548678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.549891 (+ 1213us) server_negotiation.cc:407] Connection header received
0924 20:13:39.550039 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.550042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.550107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.550160 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.550854 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551703 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.553842 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554201 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554275 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.559100 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.559107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.559113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.559149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.561050 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561600 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.561606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.561607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.561692 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.562018 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.562022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.562024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.562297 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.562385 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.563193 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.563356 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:39.565124 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.546239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.29.83.130:43111)
0924 20:13:39.548301 (+ 2062us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.548316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.548479 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.549443 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.549446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.549954 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.550686 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.550701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551861 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.551864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.553704 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.553714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554485 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.554977 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.555802 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.555818 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.559337 (+ 3519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.561386 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.561392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.561680 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.562015 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.562018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.562020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.562106 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.562602 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.562605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.562866 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.563652 (+ 786us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.564094 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1941,"mutex_wait_us":604,"spinlock_wait_cycles":9600,"thread_start_us":1475,"threads_started":1}
I20260924 20:13:39.565142 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.548193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.29.83.130:54553)
0924 20:13:39.549798 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.549812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.549923 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.550286 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.550288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.550458 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.550680 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.550701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551861 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.551864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.553665 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.553676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554485 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.554856 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.555910 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.555924 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.558934 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.561188 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.561200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.561482 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.561827 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.561835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.561917 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.562456 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.562466 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.562868 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.563706 (+ 838us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.564022 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":480,"spinlock_wait_cycles":10368,"thread_start_us":69,"threads_started":1}
I20260924 20:13:39.565814 4772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.546251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:48783)
0924 20:13:39.547160 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.547214 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.548383 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.549218 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.549238 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.549895 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.550679 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.550702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.551912 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.551916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.553675 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.553689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.554559 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.554566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.554952 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.555754 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.555789 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.558072 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.560915 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.560922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.560943 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.561336 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.561668 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.561671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.561673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.561902 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.562876 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.562879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.562950 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.564192 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.564315 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":398,"spinlock_wait_cycles":17920,"thread_start_us":132,"threads_started":1}
I20260924 20:13:39.566577 4768 heartbeater.cc:347] Connected to a master server at 127.29.83.188:34929
I20260924 20:13:39.566583 4769 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:39.566696 4770 heartbeater.cc:347] Connected to a master server at 127.29.83.189:39599
I20260924 20:13:39.566843 4768 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.566841 4769 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.566841 4770 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.567461 4768 heartbeater.cc:511] Master 127.29.83.188:34929 requested a full tablet report, sending...
I20260924 20:13:39.567462 4769 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
I20260924 20:13:39.567638 4770 heartbeater.cc:511] Master 127.29.83.189:39599 requested a full tablet report, sending...
I20260924 20:13:39.568754 3669 ts_manager.cc:194] Re-registered known tserver with Master: 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:39.568823 4269 ts_manager.cc:194] Registered new tserver with Master: 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:39.569188 4452 ts_manager.cc:194] Registered new tserver with Master: 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:39.569519 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:48783
W20260924 20:13:39.680523 4775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:39.680876 4775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:39.680946 4775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:13:39.683449 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.679262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.29.83.188:34929)
0924 20:13:39.679399 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.679406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.679539 (+ 133us) server_negotiation.cc:407] Connection header received
0924 20:13:39.679721 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.679724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.679780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.679868 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.680151 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.680645 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.682225 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.682913 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.683010 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.683141 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.683287 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.683346 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:13:39.683491 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.679172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59610)
0924 20:13:39.679437 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.679453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.679602 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.679903 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.679906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.679923 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.680034 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.680039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.680766 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.680770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.682025 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.682039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.683049 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.683052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.683159 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.683187 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.683264 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.683344 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260924 20:13:39.684293 4462 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 20:13:39.684844 4491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.684996 4491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: LEARNER
W20260924 20:13:39.686124 4775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:39.686175 4775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:39.686273 4775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:39.691916 4775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:45533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=40681
--tserver_master_addrs=127.29.83.190:41729,127.29.83.189:39599,127.29.83.188:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:39.692962 4775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:39.694240 4775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:13:39.704442 4775 server_base.cc:1034] running on GCE node
W20260924 20:13:39.704386 4784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.704377 4781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:39.704730 4782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:39.705121 4775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:39.705868 4775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:39.707048 4775 hybrid_clock.cc:648] HybridClock initialized: now 1790280819707024 us; error 46 us; skew 500 ppm
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:39.709985 4775 init.cc:377] Logged in from keytab as oryx/127.29.83.131@KRBTEST.COM (short username oryx)
I20260924 20:13:39.710914 4775 webserver.cc:460] Webserver started at http://127.29.83.131:40681/ using document root <none> and password file <none>
I20260924 20:13:39.711473 4775 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:39.711606 4775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:39.715485 4775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:13:39.717664 4791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.718981 4775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:39.719118 4775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "9331c8ccabea425db52bd9da870468bb"
format_stamp: "Formatted at 2026-09-24 20:13:38 on dist-test-slave-52fr"
server_key: "0c61b1e8aeef700644ecb64a3d40a990"
server_key_iv: "8d34751d95eb2db23b4f4ba7044d8698"
server_key_version: "encryptionkey@0"
I20260924 20:13:39.719691 4775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:39.739185 4775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:39.743188 4775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:39.743352 4775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:39.743904 4775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:39.744721 4775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:39.744837 4775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.744884 4775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:39.744904 4775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:39.753845 4775 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:45533
I20260924 20:13:39.753867 4904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:45533 every 8 connection(s)
I20260924 20:13:39.755013 4775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:39.761219 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 4775
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.188@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.189@KRBTEST.COM
Sep 24 20:13:39 dist-test-slave-52fr krb5kdc[3624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280819, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:39.773494 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.758561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:36869 (local address 127.29.83.188:34929)
0924 20:13:39.758705 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.758709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.758778 (+ 69us) server_negotiation.cc:407] Connection header received
0924 20:13:39.759661 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.759664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.759720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.759803 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.761665 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.762497 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764007 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.764353 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764451 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.768270 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.768274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.768276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.768301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.770385 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.771568 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.771573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.771576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.771687 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.772135 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.772140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.772142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.772379 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.772481 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.773216 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.773337 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:39.775071 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.758379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.29.83.131:36869)
0924 20:13:39.758674 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.758691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.759442 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.760083 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.760093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.760713 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.761470 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.761491 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.762655 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.762659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.763871 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.763881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.764775 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.764909 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.765799 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.765813 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.768064 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.770504 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.770511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.771056 (+ 545us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.771434 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.771804 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.771808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.771811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.772018 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.772537 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.772550 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.772924 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.773154 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.773528 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":282,"spinlock_wait_cycles":10496,"thread_start_us":94,"threads_started":1}
I20260924 20:13:39.776486 4909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.757439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.29.83.131:46347)
0924 20:13:39.757976 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.758023 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.759442 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.760083 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.760093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.760713 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.761626 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.761630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.762525 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.762529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.763866 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.763874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.764595 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.764833 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.765659 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.765681 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.770152 (+ 4471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.774715 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.774718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.774720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.774991 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.775324 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.775327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.775329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.775412 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.775870 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.775873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.775964 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.776073 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.776237 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":165,"spinlock_wait_cycles":18816,"thread_start_us":114,"threads_started":1}
I20260924 20:13:39.776899 4905 heartbeater.cc:347] Connected to a master server at 127.29.83.188:34929
I20260924 20:13:39.776995 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.757055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:46347 (local address 127.29.83.189:39599)
0924 20:13:39.757200 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.757204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.758124 (+ 920us) server_negotiation.cc:407] Connection header received
0924 20:13:39.759797 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.759800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.759846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.759929 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.761729 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.762388 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.763986 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.764317 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764419 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.770276 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.770279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.770283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.770308 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.774400 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.775115 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.775118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.775120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.775219 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.775513 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.775517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.775522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.775732 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.775827 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.776695 (+ 868us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.776821 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:39.777179 4905 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.777575 4907 heartbeater.cc:347] Connected to a master server at 127.29.83.189:39599
I20260924 20:13:39.777662 4907 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.777828 4905 heartbeater.cc:511] Master 127.29.83.188:34929 requested a full tablet report, sending...
I20260924 20:13:39.777858 4907 heartbeater.cc:511] Master 127.29.83.189:39599 requested a full tablet report, sending...
I20260924 20:13:39.778477 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.758881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:53967 (local address 127.29.83.190:41729)
0924 20:13:39.759006 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.759009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.759026 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:13:39.760028 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.760030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.760072 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.760160 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.763176 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.763957 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.766513 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.766864 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.766956 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.770709 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.770716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.770722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.770759 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.774595 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.777098 (+ 2503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.777102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.777104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.777203 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.777517 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.777520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.777522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.777790 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.777879 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.778217 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.778333 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:39.778945 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.758759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:53967)
0924 20:13:39.758972 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.758984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.759455 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.760177 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.760181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.760689 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:39.761471 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.761493 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.764082 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.764085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.766365 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.766379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.767125 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.767128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.767240 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.767957 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:39.767973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:39.770545 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:39.776748 (+ 6203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.776751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.776752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.776971 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.777315 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:39.777321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:39.777324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:39.777408 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:39.777937 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:39.777940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:39.778022 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.778614 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.778773 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1577,"spinlock_wait_cycles":1408,"thread_start_us":90,"threads_started":1}
I20260924 20:13:39.779184 4269 ts_manager.cc:194] Registered new tserver with Master: 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533)
I20260924 20:13:39.779616 4452 ts_manager.cc:194] Registered new tserver with Master: 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533)
I20260924 20:13:39.779808 4906 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41729
I20260924 20:13:39.779887 4906 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:39.780055 4906 heartbeater.cc:511] Master 127.29.83.190:41729 requested a full tablet report, sending...
I20260924 20:13:39.780627 3669 ts_manager.cc:194] Re-registered known tserver with Master: 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533)
I20260924 20:13:39.781407 3669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:53967
I20260924 20:13:39.781459 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.764983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.29.83.188:34929)
0924 20:13:39.765501 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.765506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.765887 (+ 381us) server_negotiation.cc:407] Connection header received
0924 20:13:39.765927 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.765930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.765975 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.766069 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.766881 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.767786 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.768942 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.769364 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.769455 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.771511 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.771525 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.771527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.771551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.778121 (+ 6570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.778725 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.778729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.778732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.778819 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.779447 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.779450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.779453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.779664 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.779795 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.780256 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.781300 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":81,"threads_started":1}
I20260924 20:13:39.782312 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.763282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.29.83.190:41729)
0924 20:13:39.763680 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.763685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.765187 (+ 1502us) server_negotiation.cc:407] Connection header received
0924 20:13:39.765296 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.765300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.765355 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.765445 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.767136 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.767904 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.771491 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.771859 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.771974 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.773709 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.773713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.773715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.773740 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.779973 (+ 6233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.780845 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.780847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.780848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.780921 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.781235 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.781240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.781241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.781463 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.781849 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.782097 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.782204 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":330,"thread_start_us":84,"threads_started":1}
I20260924 20:13:39.784402 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.764604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.29.83.189:39599)
0924 20:13:39.765729 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.765735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.765749 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:39.765816 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.765819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.765867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.765950 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:39.768523 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.769282 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.772090 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.772473 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.772579 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.774833 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:39.774837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:39.774839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:39.774862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:39.780737 (+ 5875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.781286 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.781290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.781292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.781377 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:39.783557 (+ 2180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:39.783561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:39.783563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:39.783741 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:39.783879 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.784100 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.784224 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":74,"threads_started":1}
I20260924 20:13:39.784983 3669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:39.788275 3669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:39.790875 4416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: NON_VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: true } } }
I20260924 20:13:39.791018 4416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.794356 4924 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: attempting to promote NON_VOTER 2c16523db3f84ba1821c6cbf98b063c0 to VOTER
I20260924 20:13:39.794677 4924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:39.796139 4461 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 LEARNER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:13:39.796505 4416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 20:13:39.797411 4279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Refusing update from remote peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:13:39.797917 4923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 20:13:39.798154 4926 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.799256 4279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.799782 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.799873 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.799978 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.800036 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.800858 4933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.801043 4933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:39.801385 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.801477 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:39.801187 4462 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.803490 4491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:39.803619 4491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:39.809389 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.803357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:45533 (local address 127.0.0.1:49542)
0924 20:13:39.804053 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.804070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.804154 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.805110 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.805114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.805133 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.805255 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.805263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.806638 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.806641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.807813 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.807824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.808967 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.808970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.809063 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.809089 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.809152 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.809244 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615}
I20260924 20:13:39.809466 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.803671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.29.83.131:45533)
0924 20:13:39.804520 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.804534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.804560 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:13:39.804645 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.804654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.804942 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.805421 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.805579 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.806506 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.807940 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.808838 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.808965 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.809129 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.809187 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.809300 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":124,"threads_started":1}
I20260924 20:13:39.811757 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.804805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39239 (local address 127.0.0.1:56074)
0924 20:13:39.806759 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.806772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.806856 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.807491 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.807494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.807511 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.807592 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.807596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.809015 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.809019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.810557 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.810567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.811368 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.811372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.811454 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.811478 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.811539 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.811614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"thread_start_us":87,"threads_started":1}
I20260924 20:13:39.811808 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.805400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.29.83.129:39239)
0924 20:13:39.805687 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.805703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.806953 (+ 1250us) server_negotiation.cc:407] Connection header received
0924 20:13:39.807096 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.807106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.807337 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.807496 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.807760 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.808888 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.810669 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.811239 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.811367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.811524 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.811576 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.811672 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":116,"threads_started":1}
I20260924 20:13:39.812774 4839 tablet_service.cc:1467] Processing CreateTablet for tablet d0d097ac26474144bfad4379bc84c017 (DEFAULT_TABLE table=test-table [id=56664a2865a34d89a4c41b42b8d51c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:39.813905 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d097ac26474144bfad4379bc84c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:39.815419 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.804573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34285 (local address 127.0.0.1:45552)
0924 20:13:39.805863 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.805877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.805959 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.808373 (+ 2414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.808375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.808391 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.808493 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.808499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.810790 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.810793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.812729 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.812741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.814988 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.814992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.815091 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.815114 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.815187 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.815265 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":101,"threads_started":1}
I20260924 20:13:39.815865 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.805026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.29.83.130:34285)
0924 20:13:39.807793 (+ 2767us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.807807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.807830 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:13:39.807906 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.807912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.808079 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.808260 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.808603 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.810651 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.812869 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.814658 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.814769 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.814919 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.815610 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.815732 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2649,"thread_start_us":106,"threads_started":1}
I20260924 20:13:39.815430 4565 tablet_service.cc:1467] Processing CreateTablet for tablet d0d097ac26474144bfad4379bc84c017 (DEFAULT_TABLE table=test-table [id=56664a2865a34d89a4c41b42b8d51c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:39.816478 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d097ac26474144bfad4379bc84c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:39.818879 4702 tablet_service.cc:1467] Processing CreateTablet for tablet d0d097ac26474144bfad4379bc84c017 (DEFAULT_TABLE table=test-table [id=56664a2865a34d89a4c41b42b8d51c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:39.819977 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d097ac26474144bfad4379bc84c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:39.821485 4939 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Bootstrap starting.
I20260924 20:13:39.823882 4939 tablet_bootstrap.cc:654] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:39.824743 4940 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Bootstrap starting.
I20260924 20:13:39.824744 4939 log.cc:824] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:39.827111 4940 tablet_bootstrap.cc:654] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:39.827943 4939 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: No bootstrap required, opened a new log
I20260924 20:13:39.827942 4940 log.cc:824] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:39.828405 4939 ts_tablet_manager.cc:1397] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:13:39.830032 4940 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: No bootstrap required, opened a new log
I20260924 20:13:39.830133 4942 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Bootstrap starting.
I20260924 20:13:39.830222 4940 ts_tablet_manager.cc:1397] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:13:39.831945 4942 tablet_bootstrap.cc:654] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:39.832803 4942 log.cc:824] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:39.833575 4939 raft_consensus.cc:348] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.833877 4939 raft_consensus.cc:374] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:39.833937 4939 raft_consensus.cc:729] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9331c8ccabea425db52bd9da870468bb, State: Initialized, Role: FOLLOWER
I20260924 20:13:39.834470 4939 consensus_queue.cc:260] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.835208 4940 raft_consensus.cc:348] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.835444 4940 raft_consensus.cc:374] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:39.835490 4940 raft_consensus.cc:729] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832bdff82c8a4effa0aa38001b9a33df, State: Initialized, Role: FOLLOWER
I20260924 20:13:39.835520 4939 ts_tablet_manager.cc:1428] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:13:39.835587 4942 tablet_bootstrap.cc:492] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: No bootstrap required, opened a new log
I20260924 20:13:39.835748 4906 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
I20260924 20:13:39.835758 4942 ts_tablet_manager.cc:1397] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:13:39.835934 4940 consensus_queue.cc:260] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.836833 4940 ts_tablet_manager.cc:1428] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:13:39.838841 4632 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
W20260924 20:13:39.839251 4634 tablet.cc:2369] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:39.840173 4942 raft_consensus.cc:348] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.840471 4942 raft_consensus.cc:374] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:39.840534 4942 raft_consensus.cc:729] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f1309378a415caab7efb0c3dd9d81, State: Initialized, Role: FOLLOWER
I20260924 20:13:39.841243 4942 consensus_queue.cc:260] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.842308 4942 ts_tablet_manager.cc:1428] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:13:39.842703 4769 heartbeater.cc:503] Master 127.29.83.190:41729 was elected leader, sending a full tablet report...
I20260924 20:13:39.970012 4946 raft_consensus.cc:482] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:39.970266 4946 raft_consensus.cc:504] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.971801 4946 leader_election.cc:288] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533), 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:39.975692 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.972422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:60963 (local address 127.29.83.131:45533)
0924 20:13:39.972546 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.972549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.972560 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:13:39.972605 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.972607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.972642 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.972737 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.972977 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.973479 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.974664 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.975115 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.975248 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.975305 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.975462 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.975531 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:39.975778 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.972208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:59045 (local address 127.29.83.130:34285)
0924 20:13:39.972342 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.972348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.972537 (+ 189us) server_negotiation.cc:407] Connection header received
0924 20:13:39.972666 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.972671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.972733 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.972801 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:39.973153 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.973639 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.974759 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.975253 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.975378 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.975428 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.975590 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.975660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:39.975757 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.972030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:45533 (local address 127.29.83.129:60963)
0924 20:13:39.972272 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.972287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.972393 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.972748 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.972751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.972768 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.972868 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.972874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.973588 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.973594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.974531 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.974540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.975232 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.975235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.975333 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.975356 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.975449 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.975529 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260924 20:13:39.975790 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.972065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34285 (local address 127.29.83.129:59045)
0924 20:13:39.972453 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:39.972465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:39.972579 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:39.972899 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:39.972902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:39.972919 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:39.973043 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.973048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.973754 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.973795 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.974622 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:39.974630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.975367 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.975371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:39.975471 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.975494 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:13:39.975574 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:13:39.975645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":62,"threads_started":1}
I20260924 20:13:39.976399 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d097ac26474144bfad4379bc84c017" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f1309378a415caab7efb0c3dd9d81" is_pre_election: true
I20260924 20:13:39.976743 4722 raft_consensus.cc:2454] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 0.
I20260924 20:13:39.976794 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d097ac26474144bfad4379bc84c017" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9331c8ccabea425db52bd9da870468bb" is_pre_election: true
I20260924 20:13:39.977097 4859 raft_consensus.cc:2454] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 0.
I20260924 20:13:39.977267 4520 leader_election.cc:302] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f1309378a415caab7efb0c3dd9d81, 832bdff82c8a4effa0aa38001b9a33df; no voters:
I20260924 20:13:39.977557 4946 raft_consensus.cc:2781] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:39.977641 4946 raft_consensus.cc:482] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:39.977684 4946 raft_consensus.cc:3037] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:39.978678 4946 raft_consensus.cc:504] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.979017 4946 leader_election.cc:288] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 election: Requested vote from peers 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533), 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:39.979338 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d097ac26474144bfad4379bc84c017" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9331c8ccabea425db52bd9da870468bb"
I20260924 20:13:39.979358 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d097ac26474144bfad4379bc84c017" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f1309378a415caab7efb0c3dd9d81"
I20260924 20:13:39.979477 4859 raft_consensus.cc:3037] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:39.979491 4722 raft_consensus.cc:3037] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:39.980509 4722 raft_consensus.cc:2454] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 1.
I20260924 20:13:39.980525 4859 raft_consensus.cc:2454] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 1.
I20260924 20:13:39.980777 4520 leader_election.cc:302] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f1309378a415caab7efb0c3dd9d81, 832bdff82c8a4effa0aa38001b9a33df; no voters:
I20260924 20:13:39.980955 4946 raft_consensus.cc:2781] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:39.981156 4946 raft_consensus.cc:686] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 1 LEADER]: Becoming Leader. State: Replica: 832bdff82c8a4effa0aa38001b9a33df, State: Running, Role: LEADER
I20260924 20:13:39.981508 4946 consensus_queue.cc:237] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } }
I20260924 20:13:39.985025 3669 catalog_manager.cc:5495] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df reported cstate change: term changed from 0 to 1, leader changed from <none> to 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129). New cstate: current_term: 1 leader_uuid: "832bdff82c8a4effa0aa38001b9a33df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } health_report { overall_health: HEALTHY } } }
I20260924 20:13:39.998399 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:39.995146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.29.83.129:39239)
0924 20:13:39.995263 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:39.995268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:39.995283 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:39.995397 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:39.995400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:39.995451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:39.995538 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:39.995730 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.996219 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.997029 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:39.997249 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:39.997329 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:39.997579 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:39.998084 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:39.998175 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:39.998239 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 20:13:39.998290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:13:40.004284 4859 raft_consensus.cc:1261] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Refusing update from remote peer 832bdff82c8a4effa0aa38001b9a33df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:40.004284 4722 raft_consensus.cc:1261] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Refusing update from remote peer 832bdff82c8a4effa0aa38001b9a33df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:40.005018 4946 consensus_queue.cc:1035] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:40.005251 4950 consensus_queue.cc:1035] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Connected to new peer: Peer: permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:13:40.006529 4908 tablet.cc:2369] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:40.014482 4954 mvcc.cc:205] Tried to move back new op lower bound from 7332990238730108928 to 7332990238647263232. Current Snapshot: MvccSnapshot[applied={T|T < 7332990238730108928}]
I20260924 20:13:40.018442 4955 mvcc.cc:205] Tried to move back new op lower bound from 7332990238730108928 to 7332990238647263232. Current Snapshot: MvccSnapshot[applied={T|T < 7332990238730108928}]
I20260924 20:13:40.023617 3669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:40.023793 3669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:40.027663 3669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a3d9bbbceb34ec49ed1ae6eb8be46c4: Sending DeleteTablet for 3 replicas of tablet d0d097ac26474144bfad4379bc84c017
I20260924 20:13:40.028740 4839 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d097ac26474144bfad4379bc84c017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:40 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:49542
I20260924 20:13:40.028829 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d097ac26474144bfad4379bc84c017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:40 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:45552
I20260924 20:13:40.029400 4565 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d097ac26474144bfad4379bc84c017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:40 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:56074
I20260924 20:13:40.029474 4964 tablet_replica.cc:330] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: stopping tablet replica
I20260924 20:13:40.029855 4965 tablet_replica.cc:330] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: stopping tablet replica
I20260924 20:13:40.029838 4964 raft_consensus.cc:2229] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:40.030117 4965 raft_consensus.cc:2229] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:40.030594 4963 tablet_replica.cc:330] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: stopping tablet replica
I20260924 20:13:40.031149 4964 raft_consensus.cc:2258] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:40.031172 4963 raft_consensus.cc:2229] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:40.031215 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 3637
I20260924 20:13:40.032030 4965 raft_consensus.cc:2258] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:40.032437 4963 raft_consensus.cc:2258] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:40.033528 4964 ts_tablet_manager.cc:1905] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:40.034040 4963 ts_tablet_manager.cc:1905] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:40.036410 4964 ts_tablet_manager.cc:1918] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:40.036505 4964 log.cc:1196] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/d0d097ac26474144bfad4379bc84c017
I20260924 20:13:40.036670 4963 ts_tablet_manager.cc:1918] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:40.036767 4963 log.cc:1196] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/d0d097ac26474144bfad4379bc84c017
I20260924 20:13:40.036868 4964 ts_tablet_manager.cc:1939] T d0d097ac26474144bfad4379bc84c017 P 701f1309378a415caab7efb0c3dd9d81: Deleting consensus metadata
I20260924 20:13:40.037073 4963 ts_tablet_manager.cc:1939] T d0d097ac26474144bfad4379bc84c017 P 9331c8ccabea425db52bd9da870468bb: Deleting consensus metadata
W20260924 20:13:40.040839 4657 connection.cc:535] server connection from 127.0.0.1:45552 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 20:13:40.041267 4792 connection.cc:535] server connection from 127.0.0.1:49542 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:13:40.045346 4965 ts_tablet_manager.cc:1905] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:40.047613 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:40.030418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.29.83.189:39599)
0924 20:13:40.030769 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:40.030775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:40.038338 (+ 7563us) server_negotiation.cc:407] Connection header received
0924 20:13:40.038492 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:40.038495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:40.038554 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:40.038645 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:40.039364 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.039895 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.040699 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.041037 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.041148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:40.043436 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:40.043441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:40.043443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:40.043472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:40.045202 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.046229 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.046233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.046235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.046320 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.046685 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.046688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.046700 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.046923 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:40.047026 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:40.047345 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 20:13:40.047460 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260924 20:13:40.048352 4965 ts_tablet_manager.cc:1918] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:40.048501 4965 log.cc:1196] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/d0d097ac26474144bfad4379bc84c017
I20260924 20:13:40.048878 4965 ts_tablet_manager.cc:1939] T d0d097ac26474144bfad4379bc84c017 P 832bdff82c8a4effa0aa38001b9a33df: Deleting consensus metadata
W20260924 20:13:40.050024 4520 connection.cc:441] server connection from 127.0.0.1:56074 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56074 (request call id 1) could send its response
I20260924 20:13:40.054623 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:40.045476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.29.83.189:39599)
0924 20:13:40.045709 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:40.045714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:40.045809 (+ 95us) server_negotiation.cc:407] Connection header received
0924 20:13:40.045981 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:40.045984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:40.046028 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:40.046080 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:40.046889 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.047634 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.048769 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.049091 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.049167 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:40.051002 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:40.051006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:40.051009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:40.051035 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:40.052769 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.053261 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.053265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.053267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.053345 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.053627 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.053630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.053632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.053869 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:40.053974 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:40.054342 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 20:13:40.054460 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260924 20:13:40.058255 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:40.030739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.29.83.188:34929)
0924 20:13:40.030852 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:40.030855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:40.048908 (+ 18053us) server_negotiation.cc:407] Connection header received
0924 20:13:40.049061 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:40.049065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:40.049119 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:40.049172 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:40.050451 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.051198 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.052289 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.052644 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.052727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:40.054293 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:40.054296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:40.054299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:40.054323 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:40.056345 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.056850 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.056854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.056856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.056941 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.057232 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.057235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.057237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.057451 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:40.057589 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:40.057892 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 20:13:40.058135 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:13:40.060806 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:40.045586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.29.83.188:34929)
0924 20:13:40.045834 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:40.045837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:40.051509 (+ 5672us) server_negotiation.cc:407] Connection header received
0924 20:13:40.051739 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:40.051742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:40.051787 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:40.051833 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:40.052605 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.053357 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.054618 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:40.055000 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:40.055083 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:40.057110 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:40.057114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:40.057116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:40.057137 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:40.059364 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.059726 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.059729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.059731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.059811 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:40.060041 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:40.060044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:40.060046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:40.060237 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:40.060369 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:40.060538 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 20:13:40.060643 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260924 20:13:41.016353 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.015804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:55929)
0924 20:13:41.016105 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.016220 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
W20260924 20:13:41.016734 4769 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.018435 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.018023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:33755)
0924 20:13:41.018270 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.018350 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260924 20:13:41.018805 4632 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.019905 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.019451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:59209)
0924 20:13:41.019753 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.019820 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
W20260924 20:13:41.020179 4906 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.528716 4977 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4)
I20260924 20:13:41.528909 4977 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:41.529412 4978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4)
I20260924 20:13:41.529592 4978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:41.530179 4977 leader_election.cc:288] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929)
I20260924 20:13:41.530900 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.530348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33328)
0924 20:13:41.530704 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.530788 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":60,"threads_started":1}
W20260924 20:13:41.531095 4255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.531263 4978 leader_election.cc:288] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), f7872da034d94c15a770e672e8885085 (127.29.83.189:39599)
I20260924 20:13:41.531643 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.530646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33342)
0924 20:13:41.531065 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.531487 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":54,"threads_started":1}
W20260924 20:13:41.531879 4438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.532195 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.531921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33354)
0924 20:13:41.532065 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.532126 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 20:13:41.532331 4255 leader_election.cc:334] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.533004 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.532764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33364)
0924 20:13:41.532869 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.532915 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:13:41.533531 4438 leader_election.cc:334] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.535441 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.531057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:39599 (local address 127.0.0.1:42660)
0924 20:13:41.531303 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.531317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:41.531421 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:41.531696 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:41.531699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:41.531716 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:41.531866 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:41.531874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.533022 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.533028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:41.534150 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:41.534159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.535009 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.535013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:41.535100 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:41.535163 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 20:13:41.535234 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:13:41.535306 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260924 20:13:41.535486 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.531133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.29.83.189:39599)
0924 20:13:41.531476 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:41.531480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:41.531493 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:13:41.531535 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:41.531541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:41.531588 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:41.531964 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:41.532061 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.532915 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.534271 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.534903 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.534982 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:41.535028 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:41.535273 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:13:41.535343 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":89,"threads_started":1}
I20260924 20:13:41.535633 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.530502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:34929 (local address 127.0.0.1:59648)
0924 20:13:41.530868 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.530883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:41.531488 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:41.532091 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:41.532093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:41.532107 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:41.532167 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:41.532172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.533317 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.533325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:41.534419 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:41.534427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.535197 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.535207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:41.535322 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:41.535343 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:13:41.535427 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:13:41.535485 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":70,"threads_started":1}
I20260924 20:13:41.535684 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.530643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.29.83.188:34929)
0924 20:13:41.530849 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:41.530853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:41.530953 (+ 100us) server_negotiation.cc:407] Connection header received
0924 20:13:41.531619 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:41.531637 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:41.531692 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:41.531742 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:41.532276 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.533195 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.534585 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:41.535063 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:41.535170 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:41.535238 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:41.535452 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 20:13:41.535540 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":226,"server-negotiator.queue_time_us":134,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260924 20:13:41.536396 4279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c16523db3f84ba1821c6cbf98b063c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7872da034d94c15a770e672e8885085" is_pre_election: true
I20260924 20:13:41.536401 4462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7872da034d94c15a770e672e8885085" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c16523db3f84ba1821c6cbf98b063c0" is_pre_election: true
I20260924 20:13:41.536612 4279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c16523db3f84ba1821c6cbf98b063c0 in term 1.
I20260924 20:13:41.536609 4462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7872da034d94c15a770e672e8885085 in term 1.
I20260924 20:13:41.536974 4439 leader_election.cc:302] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c16523db3f84ba1821c6cbf98b063c0, f7872da034d94c15a770e672e8885085; no voters: 5a3d9bbbceb34ec49ed1ae6eb8be46c4
I20260924 20:13:41.536962 4256 leader_election.cc:302] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c16523db3f84ba1821c6cbf98b063c0, f7872da034d94c15a770e672e8885085; no voters: 5a3d9bbbceb34ec49ed1ae6eb8be46c4
I20260924 20:13:41.537190 4977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:13:41.537264 4977 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4)
I20260924 20:13:41.537321 4977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:13:41.537854 4978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:13:41.537916 4978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a3d9bbbceb34ec49ed1ae6eb8be46c4)
I20260924 20:13:41.537955 4978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:13:41.538633 4977 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:41.539043 4977 leader_election.cc:288] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 election: Requested vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929)
I20260924 20:13:41.539243 4978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:41.539390 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.539173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33376)
0924 20:13:41.539268 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.539323 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 20:13:41.539608 4978 leader_election.cc:288] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), f7872da034d94c15a770e672e8885085 (127.29.83.189:39599)
I20260924 20:13:41.539775 4462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7872da034d94c15a770e672e8885085" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c16523db3f84ba1821c6cbf98b063c0"
I20260924 20:13:41.539875 4279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c16523db3f84ba1821c6cbf98b063c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7872da034d94c15a770e672e8885085"
I20260924 20:13:41.539948 4462 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f7872da034d94c15a770e672e8885085 in current term 2: Already voted for candidate 2c16523db3f84ba1821c6cbf98b063c0 in this term.
I20260924 20:13:41.540045 4279 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c16523db3f84ba1821c6cbf98b063c0 in current term 2: Already voted for candidate f7872da034d94c15a770e672e8885085 in this term.
I20260924 20:13:41.540143 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.539862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33388)
0924 20:13:41.540000 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.540048 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 20:13:41.540225 4255 leader_election.cc:334] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.540253 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.539688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33380)
0924 20:13:41.539926 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.540057 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 20:13:41.540371 4256 leader_election.cc:302] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7872da034d94c15a770e672e8885085; no voters: 2c16523db3f84ba1821c6cbf98b063c0, 5a3d9bbbceb34ec49ed1ae6eb8be46c4
I20260924 20:13:41.540521 4977 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:13:41.540853 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:41.540618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33404)
0924 20:13:41.540773 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:41.540810 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260924 20:13:41.540952 4438 leader_election.cc:334] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:41.541057 4438 leader_election.cc:302] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c16523db3f84ba1821c6cbf98b063c0; no voters: 5a3d9bbbceb34ec49ed1ae6eb8be46c4, f7872da034d94c15a770e672e8885085
I20260924 20:13:41.541185 4978 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:13:42.017743 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:42.017344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:58503)
0924 20:13:42.017586 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:42.017647 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260924 20:13:42.019757 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:42.019341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:58275)
0924 20:13:42.019586 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:42.019672 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260924 20:13:42.021123 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:42.020658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:34135)
0924 20:13:42.021000 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:42.021050 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":81,"threads_started":1}
I20260924 20:13:43.019308 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.018777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:42125)
0924 20:13:43.019105 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.019186 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260924 20:13:43.021202 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.020708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:44365)
0924 20:13:43.021013 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.021097 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260924 20:13:43.022243 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.021799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:49501)
0924 20:13:43.022031 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.022145 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260924 20:13:43.652254 4993 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:43.652386 4993 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:43.652822 4993 leader_election.cc:288] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929)
I20260924 20:13:43.653296 4462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7872da034d94c15a770e672e8885085" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c16523db3f84ba1821c6cbf98b063c0" is_pre_election: true
I20260924 20:13:43.653378 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.653010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33666)
0924 20:13:43.653223 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.653291 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260924 20:13:43.653467 4462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7872da034d94c15a770e672e8885085 in term 2.
I20260924 20:13:43.653870 4256 leader_election.cc:302] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c16523db3f84ba1821c6cbf98b063c0, f7872da034d94c15a770e672e8885085; no voters:
I20260924 20:13:43.654083 4993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 20:13:43.654160 4993 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:43.654175 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.653957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33680)
0924 20:13:43.654069 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.654121 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 20:13:43.654217 4993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 2 FOLLOWER]: Advancing to term 3
W20260924 20:13:43.654328 4255 leader_election.cc:334] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:43.655341 4993 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:43.655722 4993 leader_election.cc:288] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 election: Requested vote from peers 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729), 2c16523db3f84ba1821c6cbf98b063c0 (127.29.83.188:34929)
I20260924 20:13:43.656059 4462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7872da034d94c15a770e672e8885085" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c16523db3f84ba1821c6cbf98b063c0"
I20260924 20:13:43.656184 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.655924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33696)
0924 20:13:43.656058 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.656104 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 20:13:43.656188 4462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:13:43.656853 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.656652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33710)
0924 20:13:43.656752 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.656798 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 20:13:43.657040 4255 leader_election.cc:334] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
I20260924 20:13:43.657274 4462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7872da034d94c15a770e672e8885085 in term 3.
I20260924 20:13:43.657569 4256 leader_election.cc:302] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c16523db3f84ba1821c6cbf98b063c0, f7872da034d94c15a770e672e8885085; no voters: 5a3d9bbbceb34ec49ed1ae6eb8be46c4
I20260924 20:13:43.657732 4993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:13:43.657943 4993 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [term 3 LEADER]: Becoming Leader. State: Replica: f7872da034d94c15a770e672e8885085, State: Running, Role: LEADER
I20260924 20:13:43.658222 4993 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } }
I20260924 20:13:43.659662 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7872da034d94c15a770e672e8885085. Latest consensus state: current_term: 3 leader_uuid: "f7872da034d94c15a770e672e8885085" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:43.659787 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:43.660094 4998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:43.662447 4998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:43.662878 4998 catalog_manager.cc:1259] Loaded cluster ID: 583f2fa7e2ba41c498e17fb0c2f98850
I20260924 20:13:43.662948 4998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:43.663467 4998 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:43.663849 4998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Loaded TSK: 0
I20260924 20:13:43.664520 4998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:13:43.698344 4269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:43.701524 4269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:43.705004 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.704710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33720)
0924 20:13:43.704852 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.704924 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 20:13:43.705204 4462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [term 3 FOLLOWER]: Refusing update from remote peer f7872da034d94c15a770e672e8885085: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 20:13:43.705789 4996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 20:13:43.705907 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.705632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.0.0.1:33732)
0924 20:13:43.705788 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.705848 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 20:13:43.706061 4255 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 -> Peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4 (127.29.83.190:41729): Couldn't send request to peer 5a3d9bbbceb34ec49ed1ae6eb8be46c4. Status: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:13:43.707369 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7872da034d94c15a770e672e8885085. Latest consensus state: current_term: 3 leader_uuid: "f7872da034d94c15a770e672e8885085" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:43.707530 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:43.708889 4993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f7872da034d94c15a770e672e8885085" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:43.709024 4993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:43.709363 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f7872da034d94c15a770e672e8885085" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5a3d9bbbceb34ec49ed1ae6eb8be46c4" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41729 } } peers { permanent_uuid: "f7872da034d94c15a770e672e8885085" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 39599 } attrs { promote: false } } peers { permanent_uuid: "2c16523db3f84ba1821c6cbf98b063c0" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 34929 } attrs { promote: false } } }
I20260924 20:13:43.709492 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c16523db3f84ba1821c6cbf98b063c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:13:43.720151 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.715422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.29.83.129:39239)
0924 20:13:43.715684 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:43.715689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:43.715702 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:13:43.715755 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:43.715757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:43.716010 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:43.716410 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:43.716525 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.717144 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.718708 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.719555 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.719689 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.719757 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:43.719939 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 20:13:43.720015 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260924 20:13:43.720142 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.715307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39239 (local address 127.0.0.1:56094)
0924 20:13:43.715408 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.715423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:43.715517 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:43.716144 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:43.716147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:43.716162 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:43.716300 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.716307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.717308 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.717311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.718562 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.718573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.719708 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.719711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.719800 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.719823 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:13:43.719921 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:13:43.719996 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":22}
I20260924 20:13:43.721004 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.715819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34285 (local address 127.0.0.1:45558)
0924 20:13:43.716114 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.716127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:43.716214 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:43.716946 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:43.716949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:43.716966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:43.717046 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.717051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.718084 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.718087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.719636 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.719647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.720558 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.720562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.720674 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.720725 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:13:43.720800 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:13:43.720878 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":167,"thread_start_us":115,"threads_started":1}
I20260924 20:13:43.721158 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.715827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:45533 (local address 127.0.0.1:49548)
0924 20:13:43.716677 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.716692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:43.716770 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:43.717190 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:43.717193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:43.717213 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:43.717306 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.717312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.718124 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.718126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.719637 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:43.719647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.720684 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.720687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:43.720806 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.720829 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:13:43.720905 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:13:43.720974 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":117,"thread_start_us":93,"threads_started":1}
I20260924 20:13:43.721076 4565 tablet_service.cc:1467] Processing CreateTablet for tablet 9515795799254c68a649f230c7cab9e8 (DEFAULT_TABLE table=test-table [id=0616581f55254c63b2a6d7eb1ccc107e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:43.721400 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.716192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.29.83.131:45533)
0924 20:13:43.716965 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:43.716969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:43.716983 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:43.717031 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:43.717034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:43.717078 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:43.717186 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:43.717396 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.717988 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.719807 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.720439 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.720548 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.720606 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:43.720926 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 20:13:43.721014 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":70,"threads_started":1}
I20260924 20:13:43.721418 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9515795799254c68a649f230c7cab9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:43.721928 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 9515795799254c68a649f230c7cab9e8 (DEFAULT_TABLE table=test-table [id=0616581f55254c63b2a6d7eb1ccc107e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:43.722086 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.716430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.29.83.130:34285)
0924 20:13:43.716659 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:43.716663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:43.716677 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:43.716740 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:43.716742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:43.716792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:43.716910 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:43.717170 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.717966 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.719809 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.720450 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.720548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.720606 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:43.721852 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0924 20:13:43.721945 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260924 20:13:43.722168 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9515795799254c68a649f230c7cab9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:43.722522 4702 tablet_service.cc:1467] Processing CreateTablet for tablet 9515795799254c68a649f230c7cab9e8 (DEFAULT_TABLE table=test-table [id=0616581f55254c63b2a6d7eb1ccc107e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:43.722783 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9515795799254c68a649f230c7cab9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:43.724229 5010 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: Bootstrap starting.
I20260924 20:13:43.725159 5012 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Bootstrap starting.
I20260924 20:13:43.725221 5010 tablet_bootstrap.cc:654] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:43.725612 5011 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Bootstrap starting.
I20260924 20:13:43.726035 5012 tablet_bootstrap.cc:654] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:43.726512 5011 tablet_bootstrap.cc:654] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:43.726601 5010 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: No bootstrap required, opened a new log
I20260924 20:13:43.726702 5010 ts_tablet_manager.cc:1397] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 20:13:43.727155 5010 raft_consensus.cc:348] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.727265 5012 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: No bootstrap required, opened a new log
I20260924 20:13:43.727288 5010 raft_consensus.cc:374] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:43.727346 5010 raft_consensus.cc:729] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832bdff82c8a4effa0aa38001b9a33df, State: Initialized, Role: FOLLOWER
I20260924 20:13:43.727349 5012 ts_tablet_manager.cc:1397] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:43.727500 5010 consensus_queue.cc:260] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.727819 5012 raft_consensus.cc:348] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.727943 5012 raft_consensus.cc:374] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:43.728008 5012 raft_consensus.cc:729] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f1309378a415caab7efb0c3dd9d81, State: Initialized, Role: FOLLOWER
I20260924 20:13:43.728142 5012 consensus_queue.cc:260] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.728212 5011 tablet_bootstrap.cc:492] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: No bootstrap required, opened a new log
I20260924 20:13:43.728303 5011 ts_tablet_manager.cc:1397] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:43.728564 5012 ts_tablet_manager.cc:1428] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:43.730716 4770 heartbeater.cc:503] Master 127.29.83.189:39599 was elected leader, sending a full tablet report...
I20260924 20:13:43.731124 5010 ts_tablet_manager.cc:1428] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 20:13:43.731304 4633 heartbeater.cc:503] Master 127.29.83.189:39599 was elected leader, sending a full tablet report...
I20260924 20:13:43.731254 5011 raft_consensus.cc:348] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.731362 5011 raft_consensus.cc:374] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:43.731395 5011 raft_consensus.cc:729] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9331c8ccabea425db52bd9da870468bb, State: Initialized, Role: FOLLOWER
I20260924 20:13:43.731556 5011 consensus_queue.cc:260] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.731936 5011 ts_tablet_manager.cc:1428] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:43.732334 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.728951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:50283)
0924 20:13:43.732159 (+ 3208us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.732226 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3131,"thread_start_us":88,"threads_started":1}
I20260924 20:13:43.732758 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.728367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:58907)
0924 20:13:43.732637 (+ 4270us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.732686 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4190,"thread_start_us":93,"threads_started":1}
I20260924 20:13:43.733438 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.733218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:53103)
0924 20:13:43.733311 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.733365 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 20:13:43.733678 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.733196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:59115)
0924 20:13:43.733531 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.733586 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260924 20:13:43.733807 4907 heartbeater.cc:503] Master 127.29.83.189:39599 was elected leader, sending a full tablet report...
I20260924 20:13:43.734318 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.732421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:60155)
0924 20:13:43.734156 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.734211 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":92,"threads_started":1}
I20260924 20:13:43.735466 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.735027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:46905)
0924 20:13:43.735365 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.735418 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
I20260924 20:13:43.768647 5016 raft_consensus.cc:482] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:43.768784 5016 raft_consensus.cc:504] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.769248 5016 leader_election.cc:288] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533), 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:43.769667 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9515795799254c68a649f230c7cab9e8" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9331c8ccabea425db52bd9da870468bb" is_pre_election: true
I20260924 20:13:43.769738 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9515795799254c68a649f230c7cab9e8" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f1309378a415caab7efb0c3dd9d81" is_pre_election: true
I20260924 20:13:43.769865 4859 raft_consensus.cc:2454] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 0.
I20260924 20:13:43.769927 4722 raft_consensus.cc:2454] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 0.
I20260924 20:13:43.770270 4520 leader_election.cc:302] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f1309378a415caab7efb0c3dd9d81, 832bdff82c8a4effa0aa38001b9a33df; no voters:
I20260924 20:13:43.770463 5016 raft_consensus.cc:2781] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:43.770522 5016 raft_consensus.cc:482] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:43.770605 5016 raft_consensus.cc:3037] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:43.771663 5016 raft_consensus.cc:504] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.772058 5016 leader_election.cc:288] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 election: Requested vote from peers 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533), 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285)
I20260924 20:13:43.772398 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9515795799254c68a649f230c7cab9e8" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9331c8ccabea425db52bd9da870468bb"
I20260924 20:13:43.772414 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9515795799254c68a649f230c7cab9e8" candidate_uuid: "832bdff82c8a4effa0aa38001b9a33df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f1309378a415caab7efb0c3dd9d81"
I20260924 20:13:43.772532 4722 raft_consensus.cc:3037] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:43.772532 4859 raft_consensus.cc:3037] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:43.773656 4859 raft_consensus.cc:2454] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 1.
I20260924 20:13:43.773655 4722 raft_consensus.cc:2454] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 832bdff82c8a4effa0aa38001b9a33df in term 1.
I20260924 20:13:43.774012 4518 leader_election.cc:302] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 701f1309378a415caab7efb0c3dd9d81, 832bdff82c8a4effa0aa38001b9a33df, 9331c8ccabea425db52bd9da870468bb; no voters:
I20260924 20:13:43.774214 5016 raft_consensus.cc:2781] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:43.774451 5016 raft_consensus.cc:686] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 1 LEADER]: Becoming Leader. State: Replica: 832bdff82c8a4effa0aa38001b9a33df, State: Running, Role: LEADER
I20260924 20:13:43.774626 5016 consensus_queue.cc:237] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } }
I20260924 20:13:43.775279 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.775007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:33471)
0924 20:13:43.775136 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.775204 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 20:13:43.776612 4268 catalog_manager.cc:5495] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df reported cstate change: term changed from 0 to 1, leader changed from <none> to 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129). New cstate: current_term: 1 leader_uuid: "832bdff82c8a4effa0aa38001b9a33df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "832bdff82c8a4effa0aa38001b9a33df" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:43.788448 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.785363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.29.83.129:39239)
0924 20:13:43.785501 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:43.785505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:43.785520 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:43.785625 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:43.785655 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:43.785699 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:43.785800 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:43.786046 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.786554 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.787423 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:43.787653 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:43.787762 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:43.788011 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:43.788080 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:43.788159 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:43.788248 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 20:13:43.788299 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:13:43.790555 4722 raft_consensus.cc:1261] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Refusing update from remote peer 832bdff82c8a4effa0aa38001b9a33df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:43.790555 4859 raft_consensus.cc:1261] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Refusing update from remote peer 832bdff82c8a4effa0aa38001b9a33df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:43.791009 5023 consensus_queue.cc:1035] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Connected to new peer: Peer: permanent_uuid: "701f1309378a415caab7efb0c3dd9d81" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:43.791182 5016 consensus_queue.cc:1035] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9331c8ccabea425db52bd9da870468bb" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:43.793386 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.793109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:54097)
0924 20:13:43.793239 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.793306 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 20:13:43.796218 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.792873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:45139)
0924 20:13:43.796081 (+ 3208us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.796140 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3132}
I20260924 20:13:43.796228 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.794663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:53615)
0924 20:13:43.796122 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.796168 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1393}
I20260924 20:13:43.797124 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.796710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.130:42127)
0924 20:13:43.796994 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.797047 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260924 20:13:43.798875 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.798556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.129:37969)
0924 20:13:43.798745 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.798801 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 20:13:43.800722 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.800413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41729 (local address 127.29.83.131:55205)
0924 20:13:43.800586 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.800651 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 20:13:43.801303 4269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:43.801465 4269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:43.805305 4269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7872da034d94c15a770e672e8885085: Sending DeleteTablet for 3 replicas of tablet 9515795799254c68a649f230c7cab9e8
I20260924 20:13:43.806080 4565 tablet_service.cc:1514] Processing DeleteTablet for tablet 9515795799254c68a649f230c7cab9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:43 UTC) from {username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:56094
I20260924 20:13:43.806233 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet 9515795799254c68a649f230c7cab9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:43 UTC) from {username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:45558
I20260924 20:13:43.806417 4839 tablet_service.cc:1514] Processing DeleteTablet for tablet 9515795799254c68a649f230c7cab9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:43 UTC) from {username='oryx', principal='oryx/127.29.83.189@KRBTEST.COM'} at 127.0.0.1:49548
I20260924 20:13:43.806453 5037 tablet_replica.cc:330] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df: stopping tablet replica
I20260924 20:13:43.806591 5037 raft_consensus.cc:2229] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:43.806744 5039 tablet_replica.cc:330] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: stopping tablet replica
I20260924 20:13:43.806785 5037 raft_consensus.cc:2258] T 9515795799254c68a649f230c7cab9e8 P 832bdff82c8a4effa0aa38001b9a33df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:43.806877 5039 raft_consensus.cc:2229] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:43.807164 5039 raft_consensus.cc:2258] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:43.807329 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 4413
I20260924 20:13:43.808881 5038 tablet_replica.cc:330] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: stopping tablet replica
I20260924 20:13:43.809027 5038 raft_consensus.cc:2229] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:43.809178 5038 raft_consensus.cc:2258] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:43.809906 5038 ts_tablet_manager.cc:1905] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:43.811057 5039 ts_tablet_manager.cc:1905] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:43.812258 5038 ts_tablet_manager.cc:1918] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:43.812354 5038 log.cc:1196] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/9515795799254c68a649f230c7cab9e8
I20260924 20:13:43.812609 5038 ts_tablet_manager.cc:1939] T 9515795799254c68a649f230c7cab9e8 P 701f1309378a415caab7efb0c3dd9d81: Deleting consensus metadata
I20260924 20:13:43.813373 4255 catalog_manager.cc:4841] TS 701f1309378a415caab7efb0c3dd9d81 (127.29.83.130:34285): tablet 9515795799254c68a649f230c7cab9e8 (table test-table [id=0616581f55254c63b2a6d7eb1ccc107e]) successfully deleted
I20260924 20:13:43.814345 5039 ts_tablet_manager.cc:1918] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:43.814442 5039 log.cc:1196] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/9515795799254c68a649f230c7cab9e8
I20260924 20:13:43.814751 5039 ts_tablet_manager.cc:1939] T 9515795799254c68a649f230c7cab9e8 P 9331c8ccabea425db52bd9da870468bb: Deleting consensus metadata
I20260924 20:13:43.815757 4253 catalog_manager.cc:4841] TS 9331c8ccabea425db52bd9da870468bb (127.29.83.131:45533): tablet 9515795799254c68a649f230c7cab9e8 (table test-table [id=0616581f55254c63b2a6d7eb1ccc107e]) successfully deleted
I20260924 20:13:43.816668 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 4639
I20260924 20:13:43.816975 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:43.816663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39239 (local address 127.0.0.1:56098)
0924 20:13:43.816806 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:43.816866 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 20:13:43.817734 4256 catalog_manager.cc:4584] TS 832bdff82c8a4effa0aa38001b9a33df (127.29.83.129:39239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9515795799254c68a649f230c7cab9e8: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39239: connect: Connection refused (error 111)
I20260924 20:13:43.823220 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 4775
I20260924 20:13:43.829698 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 4133
I20260924 20:13:43.836444 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 4239
2026-09-24T20:13:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7467 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:43 dist-test-slave-52fr krb5kdc[5042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:43 dist-test-slave-52fr krb5kdc[5042](info): set up 2 sockets
Sep 24 20:13:43 dist-test-slave-52fr krb5kdc[5042](info): commencing operation
krb5kdc: starting...
W20260924 20:13:44.898419 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:44 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab.
Entry for principal oryx/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab.
I20260924 20:13:45.025480 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38665
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab
--principal=oryx/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:45.136636 5055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:45.136914 5055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:45.136956 5055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:45.141027 5055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:45.141086 5055 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:13:45.141104 5055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:45.141122 5055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:45.141144 5055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:45.141160 5055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:45.146596 5055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38665
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.190.keytab
--principal=oryx/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:45.147748 5055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:45.148823 5055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:45.157816 5061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:45.157961 5060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.157915 5055 server_base.cc:1034] running on GCE node
W20260924 20:13:45.157809 5063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.158851 5055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:45.159960 5055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:45.161141 5055 hybrid_clock.cc:648] HybridClock initialized: now 1790280825161091 us; error 46 us; skew 500 ppm
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:45.164274 5055 init.cc:377] Logged in from keytab as oryx/127.29.83.190@KRBTEST.COM (short username oryx)
I20260924 20:13:45.165253 5055 webserver.cc:460] Webserver started at http://127.29.83.190:44699/ using document root <none> and password file <none>
I20260924 20:13:45.165905 5055 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:45.165982 5055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:45.166214 5055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:45.168341 5055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "cead387b9b9741a69dcce858a7c098a1"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "6908d23470bb736db06c8408f62918fb"
server_key_iv: "928c492f7033930b190a0a2f8445bb3e"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.168926 5055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "cead387b9b9741a69dcce858a7c098a1"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "6908d23470bb736db06c8408f62918fb"
server_key_iv: "928c492f7033930b190a0a2f8445bb3e"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.172897 5055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:13:45.175483 5070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.176792 5055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:45.176934 5055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "cead387b9b9741a69dcce858a7c098a1"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "6908d23470bb736db06c8408f62918fb"
server_key_iv: "928c492f7033930b190a0a2f8445bb3e"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.177049 5055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:45.196830 5055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:45.199893 5055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:45.200111 5055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:45.208489 5055 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38665
I20260924 20:13:45.208503 5132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38665 every 8 connection(s)
I20260924 20:13:45.209569 5055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:45.212342 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:45.212529 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5055
I20260924 20:13:45.212680 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:45.212951 30030 external_mini_cluster.cc:1428] Setting key 4322f81e5a9159479a46ae22dc0332d1
I20260924 20:13:45.219412 5133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Bootstrap starting.
I20260924 20:13:45.222051 5133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Neither blocks nor log segments found. Creating new log.
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:45.222846 5133 log.cc:824] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:45.224941 5133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: No bootstrap required, opened a new log
I20260924 20:13:45.228061 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.214378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.29.83.190:38665)
0924 20:13:45.214786 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.214804 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.214845 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:13:45.215643 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.215668 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.216092 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.216460 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.217122 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.218635 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.219521 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.220185 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.220337 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.222923 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.222930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.222947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.222979 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.225326 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.225838 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.225842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.225847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.225950 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.226237 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.226240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.226241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.226595 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.226740 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.227033 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.227328 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260924 20:13:45.228543 5133 raft_consensus.cc:348] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } }
I20260924 20:13:45.228786 5133 raft_consensus.cc:374] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:45.228847 5133 raft_consensus.cc:729] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cead387b9b9741a69dcce858a7c098a1, State: Initialized, Role: FOLLOWER
I20260924 20:13:45.229360 5133 consensus_queue.cc:260] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } }
I20260924 20:13:45.229529 5133 raft_consensus.cc:388] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:45.229596 5133 raft_consensus.cc:482] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:45.229692 5133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:45.230703 5133 raft_consensus.cc:504] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } }
I20260924 20:13:45.231040 5133 leader_election.cc:302] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cead387b9b9741a69dcce858a7c098a1; no voters:
I20260924 20:13:45.231354 5133 leader_election.cc:288] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:45.231490 5138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:45.231732 5138 raft_consensus.cc:686] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [term 1 LEADER]: Becoming Leader. State: Replica: cead387b9b9741a69dcce858a7c098a1, State: Running, Role: LEADER
I20260924 20:13:45.232050 5138 consensus_queue.cc:237] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } }
I20260924 20:13:45.232671 5133 sys_catalog.cc:564] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:45.233922 5140 sys_catalog.cc:455] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cead387b9b9741a69dcce858a7c098a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } } }
I20260924 20:13:45.234086 5140 sys_catalog.cc:458] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:45.233909 5139 sys_catalog.cc:455] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cead387b9b9741a69dcce858a7c098a1. Latest consensus state: current_term: 1 leader_uuid: "cead387b9b9741a69dcce858a7c098a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cead387b9b9741a69dcce858a7c098a1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38665 } } }
I20260924 20:13:45.234313 5139 sys_catalog.cc:458] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:45.234935 5147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:45.237553 5147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:45.242854 5147 catalog_manager.cc:1347] Generated new cluster ID: cd0222f4f4c643fab20eb2b14329064f
I20260924 20:13:45.242928 5147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:45.252146 5147 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:45.253430 5147 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:45.263007 5147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Generated new TSK 0
I20260924 20:13:45.263810 5147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
I20260924 20:13:45.297174 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38665
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:45.415130 5159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:45.415409 5159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:45.415454 5159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:45.419490 5159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:45.419559 5159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:45.419598 5159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:45.419657 5159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:45.425087 5159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:38665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:45.426224 5159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:45.427212 5159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:45.437240 5167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.437292 5159 server_base.cc:1034] running on GCE node
W20260924 20:13:45.437242 5165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:45.437235 5164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.438022 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:45.438679 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:45.439851 5159 hybrid_clock.cc:648] HybridClock initialized: now 1790280825439814 us; error 48 us; skew 500 ppm
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:45.442742 5159 init.cc:377] Logged in from keytab as oryx/127.29.83.129@KRBTEST.COM (short username oryx)
I20260924 20:13:45.443614 5159 webserver.cc:460] Webserver started at http://127.29.83.129:41647/ using document root <none> and password file <none>
I20260924 20:13:45.444126 5159 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:45.444183 5159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:45.444340 5159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:45.446364 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "a71595604fc24d139878b09af09424d2"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "5d8d1df21e3070661a7cc68e562cf852"
server_key_iv: "817ff2ce35388a2a6aa8df4232dc0f44"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.446888 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "a71595604fc24d139878b09af09424d2"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "5d8d1df21e3070661a7cc68e562cf852"
server_key_iv: "817ff2ce35388a2a6aa8df4232dc0f44"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.450621 5159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:45.453153 5174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.454636 5159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:45.454741 5159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "a71595604fc24d139878b09af09424d2"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "5d8d1df21e3070661a7cc68e562cf852"
server_key_iv: "817ff2ce35388a2a6aa8df4232dc0f44"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.454890 5159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:45.476332 5159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:45.479703 5159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:45.479934 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:45.481587 5159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:45.481637 5159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.481707 5159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:45.481740 5159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:45.493171 5159 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:44225
I20260924 20:13:45.493227 5289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:44225 every 8 connection(s)
I20260924 20:13:45.494490 5159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:45.497362 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.482718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:45311 (local address 127.29.83.190:38665)
0924 20:13:45.482844 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.482848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.483840 (+ 992us) server_negotiation.cc:407] Connection header received
0924 20:13:45.484958 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.484962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.485046 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.485143 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.486748 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.487299 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.488734 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.489174 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.489320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.493333 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.493338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.493340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.493371 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.495138 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.495764 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.495768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.495770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.495860 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.496212 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.496215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.496217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.496420 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.496532 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.497046 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.497189 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 20:13:45.498211 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.483187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.129:45311)
0924 20:13:45.483658 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.483711 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.484661 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.485335 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.485344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.485912 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.486583 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.486601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.487495 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.487510 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.488526 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.488536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.489403 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.489408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.489686 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.491141 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.491168 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.493071 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.495320 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.495329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.495349 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.495646 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.495963 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.495966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.495968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.496111 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.496529 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.496537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.496797 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.496989 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.497325 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
I20260924 20:13:45.499523 5290 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38665
I20260924 20:13:45.499665 5290 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:45.500258 5290 heartbeater.cc:511] Master 127.29.83.190:38665 requested a full tablet report, sending...
I20260924 20:13:45.502060 5086 ts_manager.cc:194] Registered new tserver with Master: a71595604fc24d139878b09af09424d2 (127.29.83.129:44225)
I20260924 20:13:45.503435 5086 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:45311
I20260924 20:13:45.504150 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5159
I20260924 20:13:45.504289 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:45.504599 30030 external_mini_cluster.cc:1428] Setting key 77a737d8341a5a4c3056eca47c06d278
I20260924 20:13:45.510102 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.501708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.129:43083)
0924 20:13:45.502032 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.502047 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.502154 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.502471 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.502473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.502704 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.502830 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.502837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.503714 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.503716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.504641 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.504653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.505324 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.505328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.505420 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.506106 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.506124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.506511 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.508419 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.508424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.508428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.508718 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.508975 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.508981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.508984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.509099 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.509560 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.509562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.509649 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.509806 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.509944 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":141,"threads_started":1}
I20260924 20:13:45.510262 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.501864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:43083 (local address 127.29.83.190:38665)
0924 20:13:45.501992 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.501996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.502123 (+ 127us) server_negotiation.cc:407] Connection header received
0924 20:13:45.502307 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.502310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.502361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.502429 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.502969 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.503510 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.504777 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.505041 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.505153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.506648 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.506652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.506655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.506684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.508309 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.508815 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.508818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.508819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.508889 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.509194 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.509197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.509198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.509448 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.509564 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.509864 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.510028 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
I20260924 20:13:45.530756 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38665
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:45.645692 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:45.646064 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:45.646144 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:45.650249 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:45.650338 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:45.650404 5299 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:45.650487 5299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:45.655998 5299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:38665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:45.657117 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:45.658249 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:45.668123 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:45.668146 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:45.668174 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.668349 5299 server_base.cc:1034] running on GCE node
I20260924 20:13:45.668949 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:45.669610 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:45.670771 5299 hybrid_clock.cc:648] HybridClock initialized: now 1790280825670751 us; error 38 us; skew 500 ppm
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:45.673520 5299 init.cc:377] Logged in from keytab as oryx/127.29.83.130@KRBTEST.COM (short username oryx)
I20260924 20:13:45.674445 5299 webserver.cc:460] Webserver started at http://127.29.83.130:41289/ using document root <none> and password file <none>
I20260924 20:13:45.675004 5299 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:45.675071 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:45.675279 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:45.677253 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "d8a08f23f9f84189b5f41a8456deacfa"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "b53220a78d4605eaa9a0f225b73f160f"
server_key_iv: "23303782f20e08655ebbc1412baea49b"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.677863 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "d8a08f23f9f84189b5f41a8456deacfa"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "b53220a78d4605eaa9a0f225b73f160f"
server_key_iv: "23303782f20e08655ebbc1412baea49b"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.681659 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:13:45.684118 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.685348 5299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:45.685436 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "d8a08f23f9f84189b5f41a8456deacfa"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "b53220a78d4605eaa9a0f225b73f160f"
server_key_iv: "23303782f20e08655ebbc1412baea49b"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.685565 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:45.702402 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:45.705515 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:45.705714 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:45.707337 5299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:45.707391 5299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.707432 5299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:45.707477 5299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:45.718809 5299 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:42965
I20260924 20:13:45.718837 5429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:42965 every 8 connection(s)
I20260924 20:13:45.719878 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:45.721937 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.708284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:45445 (local address 127.29.83.190:38665)
0924 20:13:45.708424 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.708428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.709499 (+ 1071us) server_negotiation.cc:407] Connection header received
0924 20:13:45.710682 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.710686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.710749 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.710854 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.712122 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.713048 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.714382 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.714766 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.714937 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.718023 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.718026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.718029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.718061 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.719641 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.720217 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.720238 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.720241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.720346 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.720804 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.720808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.720810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.721067 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.721163 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.721631 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.721746 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:13:45.722723 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.708815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.130:45445)
0924 20:13:45.709345 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.709388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.710394 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.711047 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.711056 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.711409 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.711936 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.711955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.713178 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.713184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.714246 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.714256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.714997 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.715001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.715191 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.715823 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.715841 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.717822 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.719756 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.719762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.719777 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.720111 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.720458 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.720461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.720463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.720675 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.721170 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.721175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.721398 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.721582 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.721900 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":145,"threads_started":1}
I20260924 20:13:45.723973 5430 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38665
I20260924 20:13:45.724155 5430 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:45.724638 5430 heartbeater.cc:511] Master 127.29.83.190:38665 requested a full tablet report, sending...
I20260924 20:13:45.725728 5087 ts_manager.cc:194] Registered new tserver with Master: d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965)
I20260924 20:13:45.726605 5087 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:45445
I20260924 20:13:45.727741 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5299
I20260924 20:13:45.727857 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:45.728142 30030 external_mini_cluster.cc:1428] Setting key 9f180a8da76c2fc0838ad80f9d153c25
I20260924 20:13:45.734068 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.726248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:38457 (local address 127.29.83.190:38665)
0924 20:13:45.726360 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.726364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.726533 (+ 169us) server_negotiation.cc:407] Connection header received
0924 20:13:45.726639 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.726642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.726705 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.726794 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.727227 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.727898 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.728958 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.729282 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.729394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.730441 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.730447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.730453 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.730485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.732024 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.732492 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.732494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.732495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.732564 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.732929 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.732938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.732940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.733244 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.733345 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.733676 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.733846 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:13:45.734071 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.726165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.130:38457)
0924 20:13:45.726454 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.726467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.726562 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.726810 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.726813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.727016 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.727107 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.727112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.728021 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.728024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.728838 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.728847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.729389 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.729392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.729461 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.729999 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.730010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.730306 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.732133 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.732139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.732144 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.732396 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.732677 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.732682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.732687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.732787 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.733354 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.733360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.733473 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.733596 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.733806 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
I20260924 20:13:45.757716 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38665
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:45.870615 5439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:45.870939 5439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:45.871016 5439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:45.875275 5439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:45.875367 5439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:45.875442 5439 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:13:45.875523 5439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:45.881137 5439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.83.190:38665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:45.882400 5439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:45.883492 5439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:45.893554 5444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:45.893697 5447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.893610 5439 server_base.cc:1034] running on GCE node
W20260924 20:13:45.893676 5445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:45.894284 5439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:45.894955 5439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:45.896147 5439 hybrid_clock.cc:648] HybridClock initialized: now 1790280825896128 us; error 29 us; skew 500 ppm
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:45.898964 5439 init.cc:377] Logged in from keytab as oryx/127.29.83.131@KRBTEST.COM (short username oryx)
I20260924 20:13:45.899957 5439 webserver.cc:460] Webserver started at http://127.29.83.131:42261/ using document root <none> and password file <none>
I20260924 20:13:45.900651 5439 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:45.900751 5439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:45.900990 5439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:45.903174 5439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "d586fc544d7c4acba89a1ea907a00acc"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "28a80c5cf57b47ff715e71c43d1f14a6"
server_key_iv: "23461c3798be86cd4363c07e91dc3a7f"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.903769 5439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "d586fc544d7c4acba89a1ea907a00acc"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "28a80c5cf57b47ff715e71c43d1f14a6"
server_key_iv: "23461c3798be86cd4363c07e91dc3a7f"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.907786 5439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:45.910872 5454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.912595 5439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:13:45.912793 5439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "d586fc544d7c4acba89a1ea907a00acc"
format_stamp: "Formatted at 2026-09-24 20:13:45 on dist-test-slave-52fr"
server_key: "28a80c5cf57b47ff715e71c43d1f14a6"
server_key_iv: "23461c3798be86cd4363c07e91dc3a7f"
server_key_version: "encryptionkey@0"
I20260924 20:13:45.912947 5439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:45.929114 5439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:45.932518 5439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:45.932756 5439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:45.934643 5439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:45.934739 5439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:45.934809 5439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:45.934839 5439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:45.946832 5439 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:32865
I20260924 20:13:45.947103 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:32865 every 8 connection(s)
I20260924 20:13:45.947846 5439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:45.949918 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.935507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:60517 (local address 127.29.83.190:38665)
0924 20:13:45.935665 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.935668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.936775 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 20:13:45.937933 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.937936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.937999 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.938090 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.939575 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.940193 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.941420 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.941653 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.941795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.945889 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.945893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.945898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.945940 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.947563 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.948163 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.948170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.948176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.948257 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.948691 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.948694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.948695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.948891 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.948971 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.949586 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.949704 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:13:45.950780 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.936092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.131:60517)
0924 20:13:45.936619 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.936664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.937653 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.938540 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.938549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.938893 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.939416 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.939431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.940339 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.940342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.941280 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.941289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.941851 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.941855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.942092 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.942948 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.942980 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.945645 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.947697 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.947703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.947720 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.948055 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.948394 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.948397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.948399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.948572 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.949033 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.949039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.949298 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.949470 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.949748 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260924 20:13:45.952121 5570 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38665
I20260924 20:13:45.952270 5570 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:45.952925 5570 heartbeater.cc:511] Master 127.29.83.190:38665 requested a full tablet report, sending...
I20260924 20:13:45.954089 5087 ts_manager.cc:194] Registered new tserver with Master: d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:45.954845 5087 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:60517
I20260924 20:13:45.955201 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5439
I20260924 20:13:45.955325 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:45.955650 30030 external_mini_cluster.cc:1428] Setting key 02822676df516dd55b745bee17353e8c
I20260924 20:13:45.957715 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:13:45.963384 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.954533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:57737 (local address 127.29.83.190:38665)
0924 20:13:45.954648 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.954652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.954845 (+ 193us) server_negotiation.cc:407] Connection header received
0924 20:13:45.955177 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.955180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.955235 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.955287 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.956149 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.956871 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.957877 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.958117 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.958203 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.959519 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.959522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.959524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.959615 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.961376 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.962030 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.962033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.962036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.962132 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.962448 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.962452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.962465 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.962665 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.962780 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.963085 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.963217 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:45.963399 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.954458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.29.83.131:57737)
0924 20:13:45.954741 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:45.954754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:45.954870 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:45.955413 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:45.955415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:45.955658 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:45.955780 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.955787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.957021 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.957024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.957707 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:45.957717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.958238 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.958242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:45.958353 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.958860 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:45.958882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:45.959372 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:45.961578 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.961583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.961586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.961903 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.962241 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:45.962244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:45.962246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:45.962343 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:45.962779 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:45.962782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:45.962892 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 20:13:45.963014 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 20:13:45.963185 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:13:45.983949 5136 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:45.977376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.29.83.190:38665)
0924 20:13:45.977531 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.977535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.977546 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:13:45.977588 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.977590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.977629 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.977719 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.978551 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.979083 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.979954 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.980181 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.980294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.983824 (+ 3530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46788: BlockingRecv error: recv got EOF from 127.0.0.1:46788 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 20:13:45 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:45.994751 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:45.985623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.29.83.190:38665)
0924 20:13:45.985794 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:45.985801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:45.986059 (+ 258us) server_negotiation.cc:407] Connection header received
0924 20:13:45.986171 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:45.986173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:45.986216 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:45.986288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:45.986948 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.987459 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.988305 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:45.988530 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:45.988667 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:45.991683 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:45.991686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:45.991688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:45.991716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:45.993051 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.993458 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.993460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.993461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.993534 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:45.993858 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:45.993861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:45.993862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:45.994043 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:45.994168 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:45.994413 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:13:45.994578 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:13:45.997933 5087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:46.001349 5087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:46.018018 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.011516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.29.83.130:42965)
0924 20:13:46.011957 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.011969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.012557 (+ 588us) server_negotiation.cc:407] Connection header received
0924 20:13:46.012800 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.012806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.012995 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.013120 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.013557 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014389 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.016308 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.016994 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017058 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.017208 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.017733 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.017855 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":103,"threads_started":1}
I20260924 20:13:46.018016 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.012335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.29.83.129:44225)
0924 20:13:46.012827 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.012833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.012854 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:13:46.012924 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.012928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.013110 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.013546 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.013689 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014755 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.016199 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.017129 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.017488 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.017733 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.017870 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":77,"threads_started":1}
I20260924 20:13:46.018265 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.011365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42965 (local address 127.0.0.1:55062)
0924 20:13:46.012460 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.012473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.012650 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.013297 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.013301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.013324 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.013444 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.013448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014510 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.014514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.016160 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.016171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.017292 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.017575 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.017596 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.017695 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.018117 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":302,"thread_start_us":89,"threads_started":1}
I20260924 20:13:46.018270 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.012163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:44225 (local address 127.0.0.1:53074)
0924 20:13:46.012484 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.012497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.012573 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.013252 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.013258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.013280 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.013410 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.013417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014951 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.014958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.016051 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.016064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.017290 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.017499 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.017558 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.017694 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.017854 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":37,"thread_start_us":88,"threads_started":1}
I20260924 20:13:46.018837 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.011365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:32865 (local address 127.0.0.1:35380)
0924 20:13:46.012028 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.012088 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.012244 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.013014 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.013020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.013041 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.013207 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.013219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014667 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.014670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.016160 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.016178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.017330 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.018422 (+ 1089us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.018447 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.018525 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.018709 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":77,"thread_start_us":105,"threads_started":1}
I20260924 20:13:46.019671 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.011533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.29.83.131:32865)
0924 20:13:46.012150 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.012163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.012188 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:13:46.012451 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.012457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.012694 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.012846 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.013331 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.014556 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.016281 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.017132 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.017241 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.017404 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.019407 (+ 2003us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.019530 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":106,"threads_started":1}
I20260924 20:13:46.021807 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 493da5006ffe428f830e0df031cebe8f (DEFAULT_TABLE table=test-table [id=5306656646cd4a4c8c6251f1286c8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.021793 5224 tablet_service.cc:1467] Processing CreateTablet for tablet 493da5006ffe428f830e0df031cebe8f (DEFAULT_TABLE table=test-table [id=5306656646cd4a4c8c6251f1286c8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.022116 5504 tablet_service.cc:1467] Processing CreateTablet for tablet 493da5006ffe428f830e0df031cebe8f (DEFAULT_TABLE table=test-table [id=5306656646cd4a4c8c6251f1286c8760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.022972 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493da5006ffe428f830e0df031cebe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.022972 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493da5006ffe428f830e0df031cebe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.023089 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493da5006ffe428f830e0df031cebe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.030125 5598 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Bootstrap starting.
I20260924 20:13:46.031311 5600 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Bootstrap starting.
I20260924 20:13:46.031459 5599 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Bootstrap starting.
I20260924 20:13:46.032428 5598 tablet_bootstrap.cc:654] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.033228 5598 log.cc:824] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:46.033427 5599 tablet_bootstrap.cc:654] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.033427 5600 tablet_bootstrap.cc:654] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.034297 5600 log.cc:824] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:46.034297 5599 log.cc:824] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:46.035908 5599 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: No bootstrap required, opened a new log
I20260924 20:13:46.036039 5600 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: No bootstrap required, opened a new log
I20260924 20:13:46.036044 5598 tablet_bootstrap.cc:492] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: No bootstrap required, opened a new log
I20260924 20:13:46.036147 5599 ts_tablet_manager.cc:1397] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:13:46.036221 5598 ts_tablet_manager.cc:1397] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 20:13:46.036221 5600 ts_tablet_manager.cc:1397] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260924 20:13:46.039220 5599 raft_consensus.cc:348] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.039464 5599 raft_consensus.cc:374] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.039517 5599 raft_consensus.cc:729] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71595604fc24d139878b09af09424d2, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.039955 5599 consensus_queue.cc:260] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.040091 5600 raft_consensus.cc:348] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.040091 5598 raft_consensus.cc:348] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.040337 5600 raft_consensus.cc:374] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.040337 5598 raft_consensus.cc:374] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.040395 5600 raft_consensus.cc:729] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.040403 5598 raft_consensus.cc:729] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d586fc544d7c4acba89a1ea907a00acc, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.040769 5290 heartbeater.cc:503] Master 127.29.83.190:38665 was elected leader, sending a full tablet report...
I20260924 20:13:46.040804 5598 consensus_queue.cc:260] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.041034 5599 ts_tablet_manager.cc:1428] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:13:46.041491 5600 consensus_queue.cc:260] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.041817 5598 ts_tablet_manager.cc:1428] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:13:46.042281 5430 heartbeater.cc:503] Master 127.29.83.190:38665 was elected leader, sending a full tablet report...
I20260924 20:13:46.043601 5570 heartbeater.cc:503] Master 127.29.83.190:38665 was elected leader, sending a full tablet report...
I20260924 20:13:46.045223 5600 ts_tablet_manager.cc:1428] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260924 20:13:46.045980 5606 raft_consensus.cc:482] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:46.046140 5606 raft_consensus.cc:504] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.047220 5606 leader_election.cc:288] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865), a71595604fc24d139878b09af09424d2 (127.29.83.129:44225)
I20260924 20:13:46.051591 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.047496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:60171 (local address 127.29.83.129:44225)
0924 20:13:46.047635 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.047640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.047679 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:13:46.047789 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.047792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.047861 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.047949 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.048221 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.048721 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.050542 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.050952 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.051085 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.051144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.051388 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.051459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:13:46.051623 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.047439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:44225 (local address 127.29.83.130:60171)
0924 20:13:46.047606 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.047621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.047695 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.047972 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.047976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.047992 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.048110 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.048115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.048833 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.048838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.050412 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.050421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.051074 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.051078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.051252 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.051282 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.051372 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.051456 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":49}
I20260924 20:13:46.051590 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.047509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:54915 (local address 127.29.83.131:32865)
0924 20:13:46.047655 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.047660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.048214 (+ 554us) server_negotiation.cc:407] Connection header received
0924 20:13:46.048329 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.048331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.048372 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.048459 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.048737 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.049288 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.050540 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.050951 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.051084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.051145 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.051394 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.051462 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:13:46.051627 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.047442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:32865 (local address 127.29.83.130:54915)
0924 20:13:46.048115 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.048126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.048249 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.048476 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.048479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.048494 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.048621 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.048625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.049423 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.049426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.050415 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.050424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.051074 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.051078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.051256 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.051289 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.051379 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.051457 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":437,"thread_start_us":128,"threads_started":1}
I20260924 20:13:46.052353 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493da5006ffe428f830e0df031cebe8f" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2" is_pre_election: true
I20260924 20:13:46.052479 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493da5006ffe428f830e0df031cebe8f" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc" is_pre_election: true
I20260924 20:13:46.052708 5244 raft_consensus.cc:2454] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.052708 5522 raft_consensus.cc:2454] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.053257 5318 leader_election.cc:302] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.053589 5606 raft_consensus.cc:2781] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:46.053676 5606 raft_consensus.cc:482] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:46.053720 5606 raft_consensus.cc:3037] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.054809 5606 raft_consensus.cc:504] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.055220 5606 leader_election.cc:288] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Requested vote from peers d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865), a71595604fc24d139878b09af09424d2 (127.29.83.129:44225)
I20260924 20:13:46.055579 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493da5006ffe428f830e0df031cebe8f" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc"
I20260924 20:13:46.055656 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493da5006ffe428f830e0df031cebe8f" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2"
I20260924 20:13:46.055773 5522 raft_consensus.cc:3037] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.055799 5244 raft_consensus.cc:3037] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.056842 5522 raft_consensus.cc:2454] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.056843 5244 raft_consensus.cc:2454] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.057188 5317 leader_election.cc:302] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d586fc544d7c4acba89a1ea907a00acc, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.057422 5606 raft_consensus.cc:2781] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:46.057654 5606 raft_consensus.cc:686] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 1 LEADER]: Becoming Leader. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Running, Role: LEADER
I20260924 20:13:46.058061 5606 consensus_queue.cc:237] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } }
I20260924 20:13:46.061169 5087 catalog_manager.cc:5495] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130). New cstate: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:46.082050 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.078381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.29.83.130:42965)
0924 20:13:46.078510 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.078514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.078527 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:13:46.078662 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.078665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.078719 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.078811 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:46.079072 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.079579 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.080465 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.080688 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.080822 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.081111 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.081622 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.081754 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.081839 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.081896 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:13:46.088338 5244 raft_consensus.cc:1261] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.088708 5522 raft_consensus.cc:1261] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.089336 5606 consensus_queue.cc:1035] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.089524 5608 consensus_queue.cc:1035] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.098351 5612 mvcc.cc:205] Tried to move back new op lower bound from 7332990263650447360 to 7332990263536537600. Current Snapshot: MvccSnapshot[applied={T|T < 7332990263650447360}]
I20260924 20:13:46.109390 5087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.109650 5087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.112435 5087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Sending DeleteTablet for 3 replicas of tablet 493da5006ffe428f830e0df031cebe8f
I20260924 20:13:46.113324 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet 493da5006ffe428f830e0df031cebe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:55062
I20260924 20:13:46.113371 5504 tablet_service.cc:1514] Processing DeleteTablet for tablet 493da5006ffe428f830e0df031cebe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:35380
I20260924 20:13:46.113349 5224 tablet_service.cc:1514] Processing DeleteTablet for tablet 493da5006ffe428f830e0df031cebe8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53074
I20260924 20:13:46.113919 5623 tablet_replica.cc:330] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: stopping tablet replica
I20260924 20:13:46.114166 5087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:13:46.114283 5623 raft_consensus.cc:2229] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 20:13:46.114598 5087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:46.114603 5623 raft_consensus.cc:2258] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.115067 5622 tablet_replica.cc:330] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: stopping tablet replica
I20260924 20:13:46.115500 5621 tablet_replica.cc:330] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: stopping tablet replica
I20260924 20:13:46.115823 5621 raft_consensus.cc:2229] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:46.116146 5621 raft_consensus.cc:2258] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.116799 5623 ts_tablet_manager.cc:1905] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.118121 5622 raft_consensus.cc:2229] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:46.118541 5622 raft_consensus.cc:2258] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.118993 5621 ts_tablet_manager.cc:1905] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.119685 5623 ts_tablet_manager.cc:1918] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.119774 5623 log.cc:1196] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/493da5006ffe428f830e0df031cebe8f
I20260924 20:13:46.119773 5364 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1464441cd4fa297de700447f82d6c (DEFAULT_TABLE table=test-table [id=4fd49f57b6104fcea739258622f2b7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.120065 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1464441cd4fa297de700447f82d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.120007 5504 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1464441cd4fa297de700447f82d6c (DEFAULT_TABLE table=test-table [id=4fd49f57b6104fcea739258622f2b7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.120070 5623 ts_tablet_manager.cc:1939] T 493da5006ffe428f830e0df031cebe8f P a71595604fc24d139878b09af09424d2: Deleting consensus metadata
I20260924 20:13:46.120239 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1464441cd4fa297de700447f82d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.121318 5074 catalog_manager.cc:4841] TS a71595604fc24d139878b09af09424d2 (127.29.83.129:44225): tablet 493da5006ffe428f830e0df031cebe8f (table test-table [id=5306656646cd4a4c8c6251f1286c8760]) successfully deleted
I20260924 20:13:46.121733 5621 ts_tablet_manager.cc:1918] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.121834 5621 log.cc:1196] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/493da5006ffe428f830e0df031cebe8f
I20260924 20:13:46.122134 5621 ts_tablet_manager.cc:1939] T 493da5006ffe428f830e0df031cebe8f P d586fc544d7c4acba89a1ea907a00acc: Deleting consensus metadata
I20260924 20:13:46.122495 5600 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Bootstrap starting.
I20260924 20:13:46.122841 5622 ts_tablet_manager.cc:1905] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.122990 5598 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Bootstrap starting.
I20260924 20:13:46.123481 5600 tablet_bootstrap.cc:654] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.123915 5598 tablet_bootstrap.cc:654] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.124497 5224 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1464441cd4fa297de700447f82d6c (DEFAULT_TABLE table=test-table [id=4fd49f57b6104fcea739258622f2b7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.124769 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1464441cd4fa297de700447f82d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.124933 5073 catalog_manager.cc:4841] TS d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865): tablet 493da5006ffe428f830e0df031cebe8f (table test-table [id=5306656646cd4a4c8c6251f1286c8760]) successfully deleted
I20260924 20:13:46.125176 5622 ts_tablet_manager.cc:1918] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.125253 5622 log.cc:1196] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/493da5006ffe428f830e0df031cebe8f
I20260924 20:13:46.125540 5622 ts_tablet_manager.cc:1939] T 493da5006ffe428f830e0df031cebe8f P d8a08f23f9f84189b5f41a8456deacfa: Deleting consensus metadata
I20260924 20:13:46.126187 5598 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: No bootstrap required, opened a new log
I20260924 20:13:46.126266 5598 ts_tablet_manager.cc:1397] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:46.126569 5074 catalog_manager.cc:4841] TS d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965): tablet 493da5006ffe428f830e0df031cebe8f (table test-table [id=5306656646cd4a4c8c6251f1286c8760]) successfully deleted
I20260924 20:13:46.126703 5598 raft_consensus.cc:348] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.126820 5598 raft_consensus.cc:374] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.126863 5598 raft_consensus.cc:729] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d586fc544d7c4acba89a1ea907a00acc, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.126995 5598 consensus_queue.cc:260] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.127226 5599 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Bootstrap starting.
I20260924 20:13:46.127252 5598 ts_tablet_manager.cc:1428] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.128103 5599 tablet_bootstrap.cc:654] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.129323 5599 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: No bootstrap required, opened a new log
I20260924 20:13:46.129406 5599 ts_tablet_manager.cc:1397] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:46.129856 5599 raft_consensus.cc:348] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.129961 5599 raft_consensus.cc:374] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.129992 5599 raft_consensus.cc:729] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71595604fc24d139878b09af09424d2, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.130020 5600 tablet_bootstrap.cc:492] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: No bootstrap required, opened a new log
I20260924 20:13:46.130097 5600 ts_tablet_manager.cc:1397] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260924 20:13:46.130102 5599 consensus_queue.cc:260] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.130332 5599 ts_tablet_manager.cc:1428] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.130573 5600 raft_consensus.cc:348] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.130677 5600 raft_consensus.cc:374] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.130710 5600 raft_consensus.cc:729] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.130833 5600 consensus_queue.cc:260] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.131103 5600 ts_tablet_manager.cc:1428] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260924 20:13:46.198853 5571 tablet.cc:2369] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:46.221017 5431 tablet.cc:2369] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:13:46.246174 5291 tablet.cc:2369] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:13:46.341717 5616 raft_consensus.cc:482] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:46.341884 5616 raft_consensus.cc:504] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.342322 5616 leader_election.cc:288] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71595604fc24d139878b09af09424d2 (127.29.83.129:44225), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.342828 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1464441cd4fa297de700447f82d6c" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc" is_pre_election: true
I20260924 20:13:46.342830 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1464441cd4fa297de700447f82d6c" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2" is_pre_election: true
I20260924 20:13:46.342963 5522 raft_consensus.cc:2454] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.343006 5244 raft_consensus.cc:2454] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.343314 5317 leader_election.cc:302] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d586fc544d7c4acba89a1ea907a00acc, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.343483 5616 raft_consensus.cc:2781] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:46.343544 5616 raft_consensus.cc:482] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:46.343586 5616 raft_consensus.cc:3037] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.344558 5616 raft_consensus.cc:504] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.344864 5616 leader_election.cc:288] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Requested vote from peers a71595604fc24d139878b09af09424d2 (127.29.83.129:44225), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.345206 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1464441cd4fa297de700447f82d6c" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc"
I20260924 20:13:46.345270 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1464441cd4fa297de700447f82d6c" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2"
I20260924 20:13:46.345314 5522 raft_consensus.cc:3037] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.345403 5244 raft_consensus.cc:3037] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.346344 5244 raft_consensus.cc:2454] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.346346 5522 raft_consensus.cc:2454] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.346650 5318 leader_election.cc:302] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d586fc544d7c4acba89a1ea907a00acc, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.346853 5616 raft_consensus.cc:2781] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:46.346946 5616 raft_consensus.cc:686] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 1 LEADER]: Becoming Leader. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Running, Role: LEADER
I20260924 20:13:46.347098 5616 consensus_queue.cc:237] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.348618 5087 catalog_manager.cc:5495] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130). New cstate: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:13:46 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280825, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.190@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:46.388447 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.379621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.29.83.190:38665)
0924 20:13:46.379798 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.379802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.380034 (+ 232us) server_negotiation.cc:407] Connection header received
0924 20:13:46.380158 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.380160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.380205 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.380274 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:46.381006 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.381586 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.382453 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.382654 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.382747 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.385131 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:46.385133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:46.385135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:46.385156 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:46.386711 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:46.387165 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:46.387168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:46.387169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:46.387239 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:46.387564 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:46.387567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:46.387568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:46.387772 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:46.387861 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.388168 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.388337 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":90}
I20260924 20:13:46.388460 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.379490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38665 (local address 127.0.0.1:46812)
0924 20:13:46.379934 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.379947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.380075 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.380312 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.380315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.380787 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:46.380905 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.380912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.381693 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.381696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.382359 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.382368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.382805 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.382816 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.382883 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.383427 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:46.383438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:46.384964 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:46.386820 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:46.386829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:46.386834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:46.387066 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:46.387348 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:46.387351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:46.387352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:46.387451 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:46.387893 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:46.387899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:46.388009 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.388113 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.388258 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":177,"threads_started":1}
I20260924 20:13:46.392988 5087 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:46812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:13:46.393782 5087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:46.398685 5224 tablet_service.cc:1467] Processing CreateTablet for tablet 69bfaad86502454eb5954f50244c566a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e69d2dac0bd940199c83f81d5f7f5ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:46.398746 5504 tablet_service.cc:1467] Processing CreateTablet for tablet 69bfaad86502454eb5954f50244c566a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e69d2dac0bd940199c83f81d5f7f5ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:46.398746 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 69bfaad86502454eb5954f50244c566a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e69d2dac0bd940199c83f81d5f7f5ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:13:46.399084 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69bfaad86502454eb5954f50244c566a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.399116 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69bfaad86502454eb5954f50244c566a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.399116 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69bfaad86502454eb5954f50244c566a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.401659 5599 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: Bootstrap starting.
I20260924 20:13:46.401862 5600 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: Bootstrap starting.
I20260924 20:13:46.402484 5598 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: Bootstrap starting.
I20260924 20:13:46.402701 5599 tablet_bootstrap.cc:654] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.402698 5600 tablet_bootstrap.cc:654] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.403313 5598 tablet_bootstrap.cc:654] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.403808 5600 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: No bootstrap required, opened a new log
I20260924 20:13:46.403919 5600 ts_tablet_manager.cc:1397] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:46.403918 5599 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: No bootstrap required, opened a new log
I20260924 20:13:46.404000 5599 ts_tablet_manager.cc:1397] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:46.404330 5600 raft_consensus.cc:348] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.404414 5600 raft_consensus.cc:374] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.404435 5600 raft_consensus.cc:729] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.404544 5598 tablet_bootstrap.cc:492] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: No bootstrap required, opened a new log
I20260924 20:13:46.404492 5599 raft_consensus.cc:348] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.404567 5600 consensus_queue.cc:260] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.404599 5599 raft_consensus.cc:374] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.404611 5598 ts_tablet_manager.cc:1397] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 20:13:46.404637 5599 raft_consensus.cc:729] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71595604fc24d139878b09af09424d2, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.404784 5599 consensus_queue.cc:260] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.404838 5600 ts_tablet_manager.cc:1428] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:13:46.405035 5599 ts_tablet_manager.cc:1428] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.405030 5598 raft_consensus.cc:348] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.405052 5604 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.405129 5598 raft_consensus.cc:374] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.405175 5598 raft_consensus.cc:729] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d586fc544d7c4acba89a1ea907a00acc, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.405186 5604 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.405287 5598 consensus_queue.cc:260] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.405560 5605 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.405721 5605 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.404779 5616 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.406193 5616 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.405560 5598 ts_tablet_manager.cc:1428] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:13:46.503163 5616 raft_consensus.cc:482] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:46.503305 5616 raft_consensus.cc:504] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.503729 5616 leader_election.cc:288] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71595604fc24d139878b09af09424d2 (127.29.83.129:44225), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.504163 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69bfaad86502454eb5954f50244c566a" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2" is_pre_election: true
I20260924 20:13:46.504223 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69bfaad86502454eb5954f50244c566a" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc" is_pre_election: true
I20260924 20:13:46.504334 5244 raft_consensus.cc:2454] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.504365 5522 raft_consensus.cc:2454] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 0.
I20260924 20:13:46.504637 5317 leader_election.cc:302] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d586fc544d7c4acba89a1ea907a00acc, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.504804 5616 raft_consensus.cc:2781] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:46.504869 5616 raft_consensus.cc:482] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:46.504904 5616 raft_consensus.cc:3037] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.506130 5616 raft_consensus.cc:504] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.506462 5616 leader_election.cc:288] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Requested vote from peers a71595604fc24d139878b09af09424d2 (127.29.83.129:44225), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.506837 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69bfaad86502454eb5954f50244c566a" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71595604fc24d139878b09af09424d2"
I20260924 20:13:46.506861 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69bfaad86502454eb5954f50244c566a" candidate_uuid: "d8a08f23f9f84189b5f41a8456deacfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc"
I20260924 20:13:46.506969 5244 raft_consensus.cc:3037] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.506985 5522 raft_consensus.cc:3037] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.507845 5244 raft_consensus.cc:2454] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.507992 5522 raft_consensus.cc:2454] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8a08f23f9f84189b5f41a8456deacfa in term 1.
I20260924 20:13:46.508148 5318 leader_election.cc:302] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.508330 5616 raft_consensus.cc:2781] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:46.508430 5616 raft_consensus.cc:686] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [term 1 LEADER]: Becoming Leader. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Running, Role: LEADER
I20260924 20:13:46.508558 5616 consensus_queue.cc:237] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.509220 5606 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: TxnStatusTablet state changed. Reason: New leader d8a08f23f9f84189b5f41a8456deacfa. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.509317 5606 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:46.509598 5638 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:13:46.509853 5638 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:13:46.510303 5087 catalog_manager.cc:5495] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130). New cstate: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:46.532968 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.529588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42965 (local address 127.0.0.1:55074)
0924 20:13:46.529821 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.529838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.529969 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.530268 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.530271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.530279 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:46.530391 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.530397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.531103 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.531107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.531727 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.531735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.532182 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.532186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.532289 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.532331 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.532640 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.532654 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.532743 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.532832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 20:13:46.532963 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.529693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.29.83.130:42965)
0924 20:13:46.529842 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.529847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.529905 (+ 58us) server_negotiation.cc:407] Connection header received
0924 20:13:46.530091 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.530095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.530148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.530245 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:46.530493 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.530992 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.531855 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.532073 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.532208 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.532467 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.532544 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.532621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.532766 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.532815 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:46.535622 5522 raft_consensus.cc:1261] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.536073 5244 raft_consensus.cc:1261] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.536042 5616 consensus_queue.cc:1035] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.536551 5616 consensus_queue.cc:1035] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.537535 5605 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: TxnStatusTablet state changed. Reason: New leader d8a08f23f9f84189b5f41a8456deacfa. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.537647 5605 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.537870 5604 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: TxnStatusTablet state changed. Reason: New leader d8a08f23f9f84189b5f41a8456deacfa. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.537964 5604 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.539649 5608 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.539758 5608 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:46.539880 5608 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.539963 5608 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d8a08f23f9f84189b5f41a8456deacfa: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:13:46.540498 5604 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.540601 5604 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P a71595604fc24d139878b09af09424d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.542791 5605 tablet_replica.cc:439] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8a08f23f9f84189b5f41a8456deacfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } } }
I20260924 20:13:46.542910 5605 tablet_replica.cc:442] T 69bfaad86502454eb5954f50244c566a P d586fc544d7c4acba89a1ea907a00acc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:13:46.555846 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.551702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:35271 (local address 127.29.83.130:42965)
0924 20:13:46.551851 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.551856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.551952 (+ 96us) server_negotiation.cc:407] Connection header received
0924 20:13:46.552144 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.552147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.552213 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.552281 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:46.552769 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.553484 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.554733 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.554933 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.555015 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.555389 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.555456 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.555505 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.555624 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.555684 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260924 20:13:46.555853 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.551595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42965 (local address 127.29.83.130:35271)
0924 20:13:46.551877 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.551891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.552028 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.552460 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.552463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.552473 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:46.552577 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.552583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.553587 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.553590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.554596 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.554607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.555076 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.555080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.555184 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.555245 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.555533 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.555540 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.555609 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.555699 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260924 20:13:46.562793 5522 raft_consensus.cc:1261] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.562796 5244 raft_consensus.cc:1261] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Refusing update from remote peer d8a08f23f9f84189b5f41a8456deacfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.563146 5616 consensus_queue.cc:1035] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.563311 5606 consensus_queue.cc:1035] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.567046 5611 mvcc.cc:205] Tried to move back new op lower bound from 7332990265598070784 to 7332990264718884864. Current Snapshot: MvccSnapshot[applied={T|T < 7332990265598070784}]
I20260924 20:13:46.569269 5613 mvcc.cc:205] Tried to move back new op lower bound from 7332990265598070784 to 7332990264718884864. Current Snapshot: MvccSnapshot[applied={T|T < 7332990265598070784}]
I20260924 20:13:46.603595 5085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.603766 5085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.605800 5085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Sending DeleteTablet for 3 replicas of tablet c3f1464441cd4fa297de700447f82d6c
I20260924 20:13:46.606451 5224 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f1464441cd4fa297de700447f82d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53074
I20260924 20:13:46.606498 5504 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f1464441cd4fa297de700447f82d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:35380
I20260924 20:13:46.606632 5623 tablet_replica.cc:330] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: stopping tablet replica
I20260924 20:13:46.606662 5621 tablet_replica.cc:330] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: stopping tablet replica
I20260924 20:13:46.606731 5623 raft_consensus.cc:2229] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:46.606761 5621 raft_consensus.cc:2229] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:46.606841 5623 raft_consensus.cc:2258] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.606861 5621 raft_consensus.cc:2258] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.607527 5623 ts_tablet_manager.cc:1905] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.609175 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f1464441cd4fa297de700447f82d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:55062
I20260924 20:13:46.609480 5622 tablet_replica.cc:330] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: stopping tablet replica
I20260924 20:13:46.609613 5622 raft_consensus.cc:2229] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:46.609864 5622 raft_consensus.cc:2258] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.610460 5623 ts_tablet_manager.cc:1918] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:46.610532 5623 log.cc:1196] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/c3f1464441cd4fa297de700447f82d6c
I20260924 20:13:46.610595 5622 ts_tablet_manager.cc:1905] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.610596 5621 ts_tablet_manager.cc:1905] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.610782 5623 ts_tablet_manager.cc:1939] T c3f1464441cd4fa297de700447f82d6c P a71595604fc24d139878b09af09424d2: Deleting consensus metadata
I20260924 20:13:46.613160 5621 ts_tablet_manager.cc:1918] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:46.613160 5622 ts_tablet_manager.cc:1918] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:46.613232 5622 log.cc:1196] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/c3f1464441cd4fa297de700447f82d6c
I20260924 20:13:46.613232 5621 log.cc:1196] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/c3f1464441cd4fa297de700447f82d6c
I20260924 20:13:46.613485 5622 ts_tablet_manager.cc:1939] T c3f1464441cd4fa297de700447f82d6c P d8a08f23f9f84189b5f41a8456deacfa: Deleting consensus metadata
I20260924 20:13:46.613485 5621 ts_tablet_manager.cc:1939] T c3f1464441cd4fa297de700447f82d6c P d586fc544d7c4acba89a1ea907a00acc: Deleting consensus metadata
I20260924 20:13:46.614050 5074 catalog_manager.cc:4841] TS a71595604fc24d139878b09af09424d2 (127.29.83.129:44225): tablet c3f1464441cd4fa297de700447f82d6c (table test-table [id=4fd49f57b6104fcea739258622f2b7cc]) successfully deleted
I20260924 20:13:46.614115 5073 catalog_manager.cc:4841] TS d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865): tablet c3f1464441cd4fa297de700447f82d6c (table test-table [id=4fd49f57b6104fcea739258622f2b7cc]) successfully deleted
I20260924 20:13:46.614734 5074 catalog_manager.cc:4841] TS d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965): tablet c3f1464441cd4fa297de700447f82d6c (table test-table [id=4fd49f57b6104fcea739258622f2b7cc]) successfully deleted
Sep 24 20:13:46 dist-test-slave-52fr krb5kdc[5042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:13:46 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:46 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:13:46.637724 5136 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:46.630580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.29.83.190:38665)
0924 20:13:46.630719 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.630725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.630935 (+ 210us) server_negotiation.cc:407] Connection header received
0924 20:13:46.631112 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.631114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.631160 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.631253 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:46.631964 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.632653 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.633587 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.633949 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.634079 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.637570 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46820: BlockingRecv error: recv got EOF from 127.0.0.1:46820 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 20:13:46 dist-test-slave-52fr krb5kdc[5042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM
I20260924 20:13:46.648425 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.639344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.29.83.190:38665)
0924 20:13:46.639461 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.639465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.639709 (+ 244us) server_negotiation.cc:407] Connection header received
0924 20:13:46.639831 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.639833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.639875 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.639941 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:46.640728 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.641222 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.642365 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.642566 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.642640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.645261 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:46.645265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:46.645267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:46.645291 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:46.646792 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:46.647223 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:46.647228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:46.647229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:46.647305 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:46.647574 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:46.647578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:46.647580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:46.647769 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:46.647906 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.648140 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.648259 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:13:46.651129 5085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:13:46.651633 5085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:46.656548 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 (DEFAULT_TABLE table=test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.656678 5504 tablet_service.cc:1467] Processing CreateTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 (DEFAULT_TABLE table=test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.656867 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e419049770c4d77bd4c18cff24c5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.656910 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e419049770c4d77bd4c18cff24c5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.658032 5224 tablet_service.cc:1467] Processing CreateTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 (DEFAULT_TABLE table=test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.658322 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e419049770c4d77bd4c18cff24c5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.659451 5598 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Bootstrap starting.
I20260924 20:13:46.660387 5598 tablet_bootstrap.cc:654] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.660660 5599 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Bootstrap starting.
I20260924 20:13:46.661535 5599 tablet_bootstrap.cc:654] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.662729 5600 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Bootstrap starting.
I20260924 20:13:46.663046 5599 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: No bootstrap required, opened a new log
I20260924 20:13:46.663136 5599 ts_tablet_manager.cc:1397] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:46.663585 5600 tablet_bootstrap.cc:654] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.663585 5598 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: No bootstrap required, opened a new log
I20260924 20:13:46.663573 5599 raft_consensus.cc:348] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.663663 5598 ts_tablet_manager.cc:1397] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 20:13:46.663679 5599 raft_consensus.cc:374] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.663708 5599 raft_consensus.cc:729] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71595604fc24d139878b09af09424d2, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.663823 5599 consensus_queue.cc:260] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.664152 5599 ts_tablet_manager.cc:1428] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.664125 5598 raft_consensus.cc:348] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.664252 5598 raft_consensus.cc:374] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.664324 5598 raft_consensus.cc:729] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d586fc544d7c4acba89a1ea907a00acc, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.664454 5598 consensus_queue.cc:260] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.664722 5600 tablet_bootstrap.cc:492] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: No bootstrap required, opened a new log
I20260924 20:13:46.664731 5598 ts_tablet_manager.cc:1428] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:13:46.664805 5600 ts_tablet_manager.cc:1397] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:46.665179 5600 raft_consensus.cc:348] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.665292 5600 raft_consensus.cc:374] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.665328 5600 raft_consensus.cc:729] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.665446 5600 consensus_queue.cc:260] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.665716 5600 ts_tablet_manager.cc:1428] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:13:46.770481 5604 raft_consensus.cc:482] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:46.770630 5604 raft_consensus.cc:504] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.771718 5604 leader_election.cc:288] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.775440 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.771970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:42947 (local address 127.29.83.130:42965)
0924 20:13:46.772127 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.772132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.772281 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:13:46.772422 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.772432 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.772473 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.772559 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.772816 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.773315 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.774460 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.774875 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.775004 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.775062 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.775242 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.775312 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:13:46.775519 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.772223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:40949 (local address 127.29.83.131:32865)
0924 20:13:46.772466 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.772472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.772487 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:46.772537 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.772539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.772574 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.772674 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:13:46.772933 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.773448 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.774487 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.774933 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.775088 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.775147 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.775312 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.775383 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260924 20:13:46.775648 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.771906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42965 (local address 127.29.83.129:42947)
0924 20:13:46.772155 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.772191 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.772311 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.772571 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.772574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.772590 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.772711 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.772716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.773446 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.773449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.774336 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.774345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.774999 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.775002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.775115 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.775133 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.775226 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.775298 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260924 20:13:46.775911 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.771943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:32865 (local address 127.29.83.129:40949)
0924 20:13:46.772277 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:46.772291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:46.772400 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:46.772684 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:46.772687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:46.772703 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:13:46.772822 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.772827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.773561 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.773567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.774367 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:46.774376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.775081 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.775085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:46.775183 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.775210 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:13:46.775295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:13:46.775363 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260924 20:13:46.776189 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e419049770c4d77bd4c18cff24c5c74" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a08f23f9f84189b5f41a8456deacfa" is_pre_election: true
I20260924 20:13:46.776397 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e419049770c4d77bd4c18cff24c5c74" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc" is_pre_election: true
I20260924 20:13:46.776473 5384 raft_consensus.cc:2454] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 0.
I20260924 20:13:46.776525 5522 raft_consensus.cc:2454] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 0.
I20260924 20:13:46.776860 5177 leader_election.cc:302] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d586fc544d7c4acba89a1ea907a00acc; no voters:
I20260924 20:13:46.777128 5604 raft_consensus.cc:2781] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:46.777206 5604 raft_consensus.cc:482] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:46.777261 5604 raft_consensus.cc:3037] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.778324 5604 raft_consensus.cc:504] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.778678 5604 leader_election.cc:288] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 election: Requested vote from peers d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.779047 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e419049770c4d77bd4c18cff24c5c74" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a08f23f9f84189b5f41a8456deacfa"
I20260924 20:13:46.779175 5384 raft_consensus.cc:3037] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.779160 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e419049770c4d77bd4c18cff24c5c74" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc"
I20260924 20:13:46.779307 5522 raft_consensus.cc:3037] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.780220 5384 raft_consensus.cc:2454] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 1.
I20260924 20:13:46.780570 5178 leader_election.cc:302] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.780722 5522 raft_consensus.cc:2454] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 1.
I20260924 20:13:46.780754 5604 raft_consensus.cc:2781] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:46.780997 5604 raft_consensus.cc:686] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 1 LEADER]: Becoming Leader. State: Replica: a71595604fc24d139878b09af09424d2, State: Running, Role: LEADER
I20260924 20:13:46.781265 5604 consensus_queue.cc:237] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.783780 5085 catalog_manager.cc:5495] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a71595604fc24d139878b09af09424d2 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "a71595604fc24d139878b09af09424d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:46.821866 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:46.817962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.29.83.129:44225)
0924 20:13:46.818260 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:46.818269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:46.818289 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:13:46.818364 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:46.818366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:46.818419 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:46.818546 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:46.818832 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.819383 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.820276 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:46.820535 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:46.820647 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:46.820948 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:46.821423 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:46.821564 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:46.821605 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 20:13:46.821661 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260924 20:13:46.824784 5522 raft_consensus.cc:1261] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Refusing update from remote peer a71595604fc24d139878b09af09424d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.824982 5384 raft_consensus.cc:1261] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Refusing update from remote peer a71595604fc24d139878b09af09424d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:46.825292 5667 consensus_queue.cc:1035] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.825568 5604 consensus_queue.cc:1035] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:46.836038 5085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.836216 5085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:46.838537 5085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Sending DeleteTablet for 3 replicas of tablet 6e419049770c4d77bd4c18cff24c5c74
I20260924 20:13:46.839228 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:55062
I20260924 20:13:46.839270 5224 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53074
I20260924 20:13:46.839427 5504 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e419049770c4d77bd4c18cff24c5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:46 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:35380
I20260924 20:13:46.839458 5623 tablet_replica.cc:330] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: stopping tablet replica
I20260924 20:13:46.839568 5623 raft_consensus.cc:2229] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:46.839609 5622 tablet_replica.cc:330] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: stopping tablet replica
I20260924 20:13:46.839746 5622 raft_consensus.cc:2229] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:46.839905 5622 raft_consensus.cc:2258] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.839915 5623 raft_consensus.cc:2258] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.840111 5621 tablet_replica.cc:330] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: stopping tablet replica
I20260924 20:13:46.840236 5621 raft_consensus.cc:2229] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:46.840397 5621 raft_consensus.cc:2258] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:46.840657 5622 ts_tablet_manager.cc:1905] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.841320 5621 ts_tablet_manager.cc:1905] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.841373 5623 ts_tablet_manager.cc:1905] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:46.843276 5622 ts_tablet_manager.cc:1918] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.843357 5622 log.cc:1196] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/6e419049770c4d77bd4c18cff24c5c74
I20260924 20:13:46.843616 5622 ts_tablet_manager.cc:1939] T 6e419049770c4d77bd4c18cff24c5c74 P d8a08f23f9f84189b5f41a8456deacfa: Deleting consensus metadata
I20260924 20:13:46.843732 5621 ts_tablet_manager.cc:1918] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.843802 5621 log.cc:1196] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/6e419049770c4d77bd4c18cff24c5c74
I20260924 20:13:46.844039 5621 ts_tablet_manager.cc:1939] T 6e419049770c4d77bd4c18cff24c5c74 P d586fc544d7c4acba89a1ea907a00acc: Deleting consensus metadata
I20260924 20:13:46.844344 5074 catalog_manager.cc:4841] TS d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965): tablet 6e419049770c4d77bd4c18cff24c5c74 (table test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]) successfully deleted
I20260924 20:13:46.844877 5073 catalog_manager.cc:4841] TS d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865): tablet 6e419049770c4d77bd4c18cff24c5c74 (table test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]) successfully deleted
I20260924 20:13:46.844928 5623 ts_tablet_manager.cc:1918] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:13:46.844993 5623 log.cc:1196] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/6e419049770c4d77bd4c18cff24c5c74
I20260924 20:13:46.845187 5623 ts_tablet_manager.cc:1939] T 6e419049770c4d77bd4c18cff24c5c74 P a71595604fc24d139878b09af09424d2: Deleting consensus metadata
I20260924 20:13:46.845614 5085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:13:46.845935 5074 catalog_manager.cc:4841] TS a71595604fc24d139878b09af09424d2 (127.29.83.129:44225): tablet 6e419049770c4d77bd4c18cff24c5c74 (table test-table [id=44eafd00ff9d45499bc803ff36a9b6fa]) successfully deleted
W20260924 20:13:46.846134 5085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:13:46.851208 5224 tablet_service.cc:1467] Processing CreateTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed (DEFAULT_TABLE table=test-table [id=4f9d230c28864c868ca168a301fd5073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.851290 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed (DEFAULT_TABLE table=test-table [id=4f9d230c28864c868ca168a301fd5073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.851385 5504 tablet_service.cc:1467] Processing CreateTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed (DEFAULT_TABLE table=test-table [id=4f9d230c28864c868ca168a301fd5073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:13:46.851514 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bfd696f74348e08efe7e5d7a5ff7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.851514 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bfd696f74348e08efe7e5d7a5ff7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.851650 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bfd696f74348e08efe7e5d7a5ff7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:46.853969 5600 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Bootstrap starting.
I20260924 20:13:46.853969 5599 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: Bootstrap starting.
I20260924 20:13:46.853994 5598 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Bootstrap starting.
I20260924 20:13:46.854823 5600 tablet_bootstrap.cc:654] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.854832 5599 tablet_bootstrap.cc:654] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.854894 5598 tablet_bootstrap.cc:654] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:46.856293 5598 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: No bootstrap required, opened a new log
I20260924 20:13:46.856379 5598 ts_tablet_manager.cc:1397] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 20:13:46.856503 5600 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: No bootstrap required, opened a new log
I20260924 20:13:46.856580 5600 ts_tablet_manager.cc:1397] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:46.856997 5598 raft_consensus.cc:348] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.857121 5598 raft_consensus.cc:374] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.857152 5598 raft_consensus.cc:729] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d586fc544d7c4acba89a1ea907a00acc, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.857136 5600 raft_consensus.cc:348] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.857264 5600 raft_consensus.cc:374] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.857307 5600 raft_consensus.cc:729] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a08f23f9f84189b5f41a8456deacfa, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.857278 5598 consensus_queue.cc:260] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.857435 5600 consensus_queue.cc:260] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.857517 5598 ts_tablet_manager.cc:1428] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.857707 5600 ts_tablet_manager.cc:1428] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:13:46.859483 5599 tablet_bootstrap.cc:492] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: No bootstrap required, opened a new log
I20260924 20:13:46.859572 5599 ts_tablet_manager.cc:1397] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 20:13:46.860049 5599 raft_consensus.cc:348] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.860189 5599 raft_consensus.cc:374] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:46.860236 5599 raft_consensus.cc:729] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71595604fc24d139878b09af09424d2, State: Initialized, Role: FOLLOWER
I20260924 20:13:46.860366 5599 consensus_queue.cc:260] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.860653 5599 ts_tablet_manager.cc:1428] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:13:46.989936 5673 raft_consensus.cc:482] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:13:46.990108 5673 raft_consensus.cc:504] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.990592 5673 leader_election.cc:288] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.991040 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bfd696f74348e08efe7e5d7a5ff7ed" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a08f23f9f84189b5f41a8456deacfa" is_pre_election: true
I20260924 20:13:46.991065 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bfd696f74348e08efe7e5d7a5ff7ed" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc" is_pre_election: true
I20260924 20:13:46.991220 5522 raft_consensus.cc:2454] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 0.
I20260924 20:13:46.991216 5384 raft_consensus.cc:2454] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 0.
I20260924 20:13:46.991501 5178 leader_election.cc:302] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.991675 5673 raft_consensus.cc:2781] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:13:46.991760 5673 raft_consensus.cc:482] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:13:46.991814 5673 raft_consensus.cc:3037] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.992746 5673 raft_consensus.cc:504] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.993093 5673 leader_election.cc:288] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 election: Requested vote from peers d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965), d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865)
I20260924 20:13:46.993403 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bfd696f74348e08efe7e5d7a5ff7ed" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a08f23f9f84189b5f41a8456deacfa"
I20260924 20:13:46.993430 5522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bfd696f74348e08efe7e5d7a5ff7ed" candidate_uuid: "a71595604fc24d139878b09af09424d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d586fc544d7c4acba89a1ea907a00acc"
I20260924 20:13:46.993530 5384 raft_consensus.cc:3037] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.993538 5522 raft_consensus.cc:3037] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:46.994467 5384 raft_consensus.cc:2454] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 1.
I20260924 20:13:46.994741 5522 raft_consensus.cc:2454] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71595604fc24d139878b09af09424d2 in term 1.
I20260924 20:13:46.994752 5178 leader_election.cc:302] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71595604fc24d139878b09af09424d2, d8a08f23f9f84189b5f41a8456deacfa; no voters:
I20260924 20:13:46.994951 5673 raft_consensus.cc:2781] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:46.995060 5673 raft_consensus.cc:686] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 1 LEADER]: Becoming Leader. State: Replica: a71595604fc24d139878b09af09424d2, State: Running, Role: LEADER
I20260924 20:13:46.995216 5673 consensus_queue.cc:237] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } }
I20260924 20:13:46.996989 5086 catalog_manager.cc:5495] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a71595604fc24d139878b09af09424d2 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "a71595604fc24d139878b09af09424d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71595604fc24d139878b09af09424d2" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260924 20:13:47.029726 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:47.026234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42965 (local address 127.29.83.129:35863)
0924 20:13:47.026559 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:47.026572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:47.026713 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:47.026983 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:47.026986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:47.026994 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:47.027101 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:47.027107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.027844 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.027849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:47.028493 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:47.028502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.028964 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.028967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:47.029064 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:47.029110 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:47.029432 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:47.029442 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 20:13:47.029506 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 20:13:47.029597 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260924 20:13:47.029723 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:47.026300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:35863 (local address 127.29.83.130:42965)
0924 20:13:47.026447 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:47.026453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:47.026662 (+ 209us) server_negotiation.cc:407] Connection header received
0924 20:13:47.026831 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:47.026836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:47.026887 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:47.026973 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:47.027221 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.027729 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.028617 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.028847 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.028960 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:47.029273 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:47.029343 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:47.029426 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:47.029515 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 20:13:47.029564 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:13:47.035701 5384 raft_consensus.cc:1261] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Refusing update from remote peer a71595604fc24d139878b09af09424d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:47.036046 5522 raft_consensus.cc:1261] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Refusing update from remote peer a71595604fc24d139878b09af09424d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:13:47.036087 5673 consensus_queue.cc:1035] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a08f23f9f84189b5f41a8456deacfa" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:47.036437 5667 consensus_queue.cc:1035] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d586fc544d7c4acba89a1ea907a00acc" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:13:47.051399 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:47.048180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:44225 (local address 127.29.83.130:44205)
0924 20:13:47.048292 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:47.048307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:47.048389 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:47.048658 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:47.048661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:47.048668 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:13:47.048767 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:47.048773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.049503 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.049509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:47.050183 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:47.050192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.050672 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.050692 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:47.050784 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:47.050817 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:13:47.051130 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:13:47.051139 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:13:47.051201 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 20:13:47.051287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 20:13:47.051399 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:47.048344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:44205 (local address 127.29.83.129:44225)
0924 20:13:47.048459 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:47.048464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:47.048478 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:47.048523 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:47.048525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:47.048568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:47.048646 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:13:47.048873 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.049395 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.050299 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:47.050556 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:47.050677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:47.050945 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:13:47.051027 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:13:47.051120 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:47.051210 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 20:13:47.051261 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:13:47.072359 5087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:47.072515 5087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:13:47.074622 5087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cead387b9b9741a69dcce858a7c098a1: Sending DeleteTablet for 3 replicas of tablet 48bfd696f74348e08efe7e5d7a5ff7ed
I20260924 20:13:47.075270 5224 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:47 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53074
I20260924 20:13:47.075296 5504 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:47 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:35380
I20260924 20:13:47.075495 5623 tablet_replica.cc:330] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: stopping tablet replica
I20260924 20:13:47.075647 5623 raft_consensus.cc:2229] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:13:47.075712 5621 tablet_replica.cc:330] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: stopping tablet replica
I20260924 20:13:47.075901 5623 raft_consensus.cc:2258] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:47.075963 5621 raft_consensus.cc:2229] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:47.076107 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bfd696f74348e08efe7e5d7a5ff7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:13:47 UTC) from {username='oryx', principal='oryx/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:55062
I20260924 20:13:47.076162 5621 raft_consensus.cc:2258] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:47.076289 5622 tablet_replica.cc:330] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: stopping tablet replica
I20260924 20:13:47.076418 5622 raft_consensus.cc:2229] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:13:47.076540 5622 raft_consensus.cc:2258] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:13:47.077013 5621 ts_tablet_manager.cc:1905] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:47.077055 5623 ts_tablet_manager.cc:1905] T 48bfd696f74348e08efe7e5d7a5ff7ed P a71595604fc24d139878b09af09424d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:47.077154 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5159
I20260924 20:13:47.077319 5622 ts_tablet_manager.cc:1905] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:13:47.079638 5621 ts_tablet_manager.cc:1918] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:47.079638 5622 ts_tablet_manager.cc:1918] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:13:47.079725 5621 log.cc:1196] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/48bfd696f74348e08efe7e5d7a5ff7ed
I20260924 20:13:47.079725 5622 log.cc:1196] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/48bfd696f74348e08efe7e5d7a5ff7ed
I20260924 20:13:47.079982 5621 ts_tablet_manager.cc:1939] T 48bfd696f74348e08efe7e5d7a5ff7ed P d586fc544d7c4acba89a1ea907a00acc: Deleting consensus metadata
I20260924 20:13:47.080008 5622 ts_tablet_manager.cc:1939] T 48bfd696f74348e08efe7e5d7a5ff7ed P d8a08f23f9f84189b5f41a8456deacfa: Deleting consensus metadata
I20260924 20:13:47.080974 5073 catalog_manager.cc:4841] TS d586fc544d7c4acba89a1ea907a00acc (127.29.83.131:32865): tablet 48bfd696f74348e08efe7e5d7a5ff7ed (table test-table [id=4f9d230c28864c868ca168a301fd5073]) successfully deleted
I20260924 20:13:47.081056 5074 catalog_manager.cc:4841] TS d8a08f23f9f84189b5f41a8456deacfa (127.29.83.130:42965): tablet 48bfd696f74348e08efe7e5d7a5ff7ed (table test-table [id=4f9d230c28864c868ca168a301fd5073]) successfully deleted
W20260924 20:13:47.086184 5074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:44225 (error 108)
I20260924 20:13:47.086395 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5299
I20260924 20:13:47.088769 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:47.088256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:44225 (local address 127.0.0.1:53100)
0924 20:13:47.088525 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:47.088619 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":52,"threads_started":1}
W20260924 20:13:47.088955 5074 catalog_manager.cc:4584] TS a71595604fc24d139878b09af09424d2 (127.29.83.129:44225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48bfd696f74348e08efe7e5d7a5ff7ed: Network error: Client connection negotiation failed: client connection to 127.29.83.129:44225: connect: Connection refused (error 111)
I20260924 20:13:47.094300 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5439
I20260924 20:13:47.100044 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5055
2026-09-24T20:13:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3261 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:47 dist-test-slave-52fr krb5kdc[5689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:47 dist-test-slave-52fr krb5kdc[5689](info): set up 2 sockets
Sep 24 20:13:47 dist-test-slave-52fr krb5kdc[5689](info): commencing operation
krb5kdc: starting...
W20260924 20:13:48.146703 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:48 dist-test-slave-52fr krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:48.269227 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41427
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:48.380236 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:48.380602 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:48.380705 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:48.384793 5702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:48.384894 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:48.384933 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:48.384970 5702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:48.385000 5702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:48.390242 5702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:48.391284 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:48.392326 5702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:48.401328 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:48.401336 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:48.401475 5702 server_base.cc:1034] running on GCE node
W20260924 20:13:48.401655 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:48.402304 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:48.403405 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:48.404567 5702 hybrid_clock.cc:648] HybridClock initialized: now 1790280828404541 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T20:13:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1315 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:13:48 dist-test-slave-52fr krb5kdc[5715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:13:48 dist-test-slave-52fr krb5kdc[5715](info): set up 2 sockets
Sep 24 20:13:48 dist-test-slave-52fr krb5kdc[5715](info): commencing operation
krb5kdc: starting...
W20260924 20:13:49.470542 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:13:49 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:13:49.597368 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:43721
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:43721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:49.707087 5728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:49.707350 5728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:49.707391 5728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:49.711395 5728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:49.711447 5728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:49.711467 5728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:49.711483 5728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:49.711498 5728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:49.716673 5728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:43721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:49.717674 5728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:49.718693 5728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:49.727576 5736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:49.727598 5733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:49.727598 5734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:49.727888 5728 server_base.cc:1034] running on GCE node
I20260924 20:13:49.728497 5728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:49.729662 5728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:49.730855 5728 hybrid_clock.cc:648] HybridClock initialized: now 1790280829730835 us; error 35 us; skew 500 ppm
Sep 24 20:13:49 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:49.742578 5728 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:49.743707 5728 webserver.cc:460] Webserver started at http://127.29.83.190:42845/ using document root <none> and password file <none>
I20260924 20:13:49.744341 5728 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:49.744436 5728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:49.744688 5728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:49.746896 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "72da6c5188344f67a7f8d97338084bb3"
format_stamp: "Formatted at 2026-09-24 20:13:49 on dist-test-slave-52fr"
server_key: "262f9192e22ff2913e1b2879f0af975a"
server_key_iv: "48cc0a1d706b7823547e8ad83615a550"
server_key_version: "encryptionkey@0"
I20260924 20:13:49.747483 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "72da6c5188344f67a7f8d97338084bb3"
format_stamp: "Formatted at 2026-09-24 20:13:49 on dist-test-slave-52fr"
server_key: "262f9192e22ff2913e1b2879f0af975a"
server_key_iv: "48cc0a1d706b7823547e8ad83615a550"
server_key_version: "encryptionkey@0"
I20260924 20:13:49.751401 5728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:13:49.753883 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:49.755097 5728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:49.755239 5728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "72da6c5188344f67a7f8d97338084bb3"
format_stamp: "Formatted at 2026-09-24 20:13:49 on dist-test-slave-52fr"
server_key: "262f9192e22ff2913e1b2879f0af975a"
server_key_iv: "48cc0a1d706b7823547e8ad83615a550"
server_key_version: "encryptionkey@0"
I20260924 20:13:49.755352 5728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:49.780268 5728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:49.783281 5728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:49.783492 5728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:49.790477 5728 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:43721
I20260924 20:13:49.790511 5795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:43721 every 8 connection(s)
I20260924 20:13:49.791424 5728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:49.794212 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:13:49.794811 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5728
I20260924 20:13:49.794951 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:13:49.795241 30030 external_mini_cluster.cc:1428] Setting key 0c05bbb8c805d8bb14310253da85bd70
I20260924 20:13:49.801044 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Bootstrap starting.
I20260924 20:13:49.803748 5796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Neither blocks nor log segments found. Creating new log.
I20260924 20:13:49.804569 5796 log.cc:824] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:13:49 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:49.806916 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: No bootstrap required, opened a new log
I20260924 20:13:49.810420 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:49.796669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.29.83.190:43721)
0924 20:13:49.797105 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:49.797124 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:49.797188 (+ 64us) server_negotiation.cc:407] Connection header received
0924 20:13:49.798054 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:49.798079 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:49.798508 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:49.798878 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:49.799525 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:49.800997 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:49.801992 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:49.802691 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:49.802848 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:49.805668 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:49.805673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:49.805687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:49.805712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:49.807481 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:49.807959 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:49.807963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:49.807967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:49.808074 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:49.808359 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:49.808361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:49.808362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:49.808748 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:49.808891 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:49.809270 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 20:13:49.809580 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260924 20:13:49.811122 5796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:49.811375 5796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:13:49.811455 5796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da6c5188344f67a7f8d97338084bb3, State: Initialized, Role: FOLLOWER
I20260924 20:13:49.811942 5796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:49.812096 5796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:49.812153 5796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:49.812237 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:13:49.813411 5796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:49.813819 5796 leader_election.cc:302] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72da6c5188344f67a7f8d97338084bb3; no voters:
I20260924 20:13:49.814186 5796 leader_election.cc:288] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:13:49.814318 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:13:49.814576 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 72da6c5188344f67a7f8d97338084bb3, State: Running, Role: LEADER
I20260924 20:13:49.814947 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:49.815595 5796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:49.816728 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72da6c5188344f67a7f8d97338084bb3. Latest consensus state: current_term: 1 leader_uuid: "72da6c5188344f67a7f8d97338084bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } } }
I20260924 20:13:49.816857 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:49.817183 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72da6c5188344f67a7f8d97338084bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } } }
I20260924 20:13:49.817272 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:49.817378 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:49.820637 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:49.826687 5810 catalog_manager.cc:1347] Generated new cluster ID: 9fdfcbb3be95402f85bd00966b9ce45f
I20260924 20:13:49.826792 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:49.837838 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:13:49.838832 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:49.845502 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Generated new TSK 0
I20260924 20:13:49.846284 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:13:49.878834 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:50.011176 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:50.011476 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:50.011520 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:50.015614 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:50.015686 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:50.015758 5822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:50.021385 5822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:50.022539 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:50.023541 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:50.033826 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.033937 5830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.033830 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.034446 5822 server_base.cc:1034] running on GCE node
I20260924 20:13:50.034976 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:50.035808 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:50.037020 5822 hybrid_clock.cc:648] HybridClock initialized: now 1790280830037008 us; error 29 us; skew 500 ppm
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:50.040006 5822 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:13:50.040867 5822 webserver.cc:460] Webserver started at http://127.29.83.129:32829/ using document root <none> and password file <none>
I20260924 20:13:50.041458 5822 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:50.041520 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:50.041731 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:50.043804 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "4dd1c4afd5b443be8615787b971b1b91"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "c59dccfd1f1db84b79281a424b600df0"
server_key_iv: "d79d668a0b640707614ad832acf3305a"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.044374 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "4dd1c4afd5b443be8615787b971b1b91"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "c59dccfd1f1db84b79281a424b600df0"
server_key_iv: "d79d668a0b640707614ad832acf3305a"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.048238 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:13:50.050812 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.052084 5822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:13:50.052214 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "4dd1c4afd5b443be8615787b971b1b91"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "c59dccfd1f1db84b79281a424b600df0"
server_key_iv: "d79d668a0b640707614ad832acf3305a"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.052328 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:50.070281 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:50.073726 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:50.074013 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:50.074656 5822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:50.075546 5822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:50.075640 5822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.075716 5822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:50.075766 5822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.085318 5822 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:41663
I20260924 20:13:50.085376 5950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:41663 every 8 connection(s)
I20260924 20:13:50.086387 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:50.090719 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5822
I20260924 20:13:50.090838 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:13:50.091192 30030 external_mini_cluster.cc:1428] Setting key efb7e6d73537926153023068614a27da
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:13:50.103200 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.088507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:58617 (local address 127.29.83.190:43721)
0924 20:13:50.088683 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:50.088688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:50.089590 (+ 902us) server_negotiation.cc:407] Connection header received
0924 20:13:50.091135 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:50.091138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:50.091203 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:50.091306 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:50.092759 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.093626 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.094960 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.095219 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.095340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.098360 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:50.098367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:50.098372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:50.098410 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:50.100393 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.101205 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.101210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.101212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.101349 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.101828 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.101833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.101837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.102109 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:50.102211 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:50.102920 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 20:13:50.103062 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:13:50.104161 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.088950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:58617)
0924 20:13:50.089428 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:50.089473 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:50.090863 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:50.091493 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:50.091502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:50.091986 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:50.092579 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.092602 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.093818 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.093821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.094824 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.094834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.095386 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.095391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.095631 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.096276 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:50.096302 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:50.098153 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:50.100599 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.100605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.100621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.101031 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.101468 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.101473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.101475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.101671 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.102272 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:50.102278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:50.102619 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 20:13:50.102802 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 20:13:50.103089 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260924 20:13:50.105849 5951 heartbeater.cc:347] Connected to a master server at 127.29.83.190:43721
I20260924 20:13:50.106154 5951 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:50.106822 5951 heartbeater.cc:511] Master 127.29.83.190:43721 requested a full tablet report, sending...
I20260924 20:13:50.108789 5760 ts_manager.cc:194] Registered new tserver with Master: 4dd1c4afd5b443be8615787b971b1b91 (127.29.83.129:41663)
I20260924 20:13:50.110159 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:58617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:13:50.120905 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:50.230432 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:50.230684 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:50.230767 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:50.234843 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:50.234902 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:50.235008 5956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:50.240427 5956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:50.241544 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:50.242672 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:50.252347 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.252359 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.252539 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.252522 5956 server_base.cc:1034] running on GCE node
I20260924 20:13:50.253160 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:50.253887 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:50.255056 5956 hybrid_clock.cc:648] HybridClock initialized: now 1790280830255024 us; error 42 us; skew 500 ppm
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:50.258164 5956 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:13:50.259197 5956 webserver.cc:460] Webserver started at http://127.29.83.130:35965/ using document root <none> and password file <none>
I20260924 20:13:50.259827 5956 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:50.259891 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:50.260108 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:50.262303 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "2fbb04ea9ae14b86a4c9654b26f0ae98"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "428cf85ffab9f8d033aa1c6a75307649"
server_key_iv: "db2c1089bcfa4cd31fa86b7136d55dc1"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.262876 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "2fbb04ea9ae14b86a4c9654b26f0ae98"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "428cf85ffab9f8d033aa1c6a75307649"
server_key_iv: "db2c1089bcfa4cd31fa86b7136d55dc1"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.266758 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:13:50.269270 5971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.270766 5956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:50.270907 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "2fbb04ea9ae14b86a4c9654b26f0ae98"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "428cf85ffab9f8d033aa1c6a75307649"
server_key_iv: "db2c1089bcfa4cd31fa86b7136d55dc1"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.271027 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:50.307960 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:50.311421 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:50.311640 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:50.312233 5956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:50.313038 5956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:50.313107 5956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.313177 5956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:50.313230 5956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.321986 5956 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:45201
I20260924 20:13:50.322024 6084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:45201 every 8 connection(s)
I20260924 20:13:50.322847 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:50.327922 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 5956
I20260924 20:13:50.328053 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:13:50.328357 30030 external_mini_cluster.cc:1428] Setting key 68a6d275d093d2fa198036405f1a5c63
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:50.337713 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.324464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:53851 (local address 127.29.83.190:43721)
0924 20:13:50.324593 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:50.324597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:50.325432 (+ 835us) server_negotiation.cc:407] Connection header received
0924 20:13:50.326861 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:50.326864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:50.326933 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:50.327036 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:50.328451 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.329071 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.330486 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.330927 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.331055 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.333640 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:50.333644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:50.333647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:50.333677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:50.335217 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.335930 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.335933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.335934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.336016 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.336464 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.336467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.336468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.336686 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:50.336771 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:50.337462 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 20:13:50.337583 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:13:50.338956 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.324875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:53851)
0924 20:13:50.325291 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:50.325331 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:50.326648 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:50.327208 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:50.327214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:50.327568 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:50.328264 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.328283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.329213 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.329217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.330355 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.330365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.331097 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.331100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.331256 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.331924 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:50.331941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:50.333459 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:50.335361 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.335370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.335391 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.335798 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.336118 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.336123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.336127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.336333 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.336804 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:50.336815 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:50.337154 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 20:13:50.337393 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 20:13:50.337820 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:13:50.340224 6085 heartbeater.cc:347] Connected to a master server at 127.29.83.190:43721
I20260924 20:13:50.340497 6085 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:50.341002 6085 heartbeater.cc:511] Master 127.29.83.190:43721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:13:50.342291 5760 ts_manager.cc:194] Registered new tserver with Master: 2fbb04ea9ae14b86a4c9654b26f0ae98 (127.29.83.130:45201)
I20260924 20:13:50.343150 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:53851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:13:50.357484 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:50.474372 6090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:50.474699 6090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:50.474786 6090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:50.479108 6090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:50.479207 6090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:50.479302 6090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:50.484840 6090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:50.486074 6090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:50.487207 6090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:50.497700 6095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.497876 6090 server_base.cc:1034] running on GCE node
W20260924 20:13:50.497741 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.497711 6098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.498515 6090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:50.499276 6090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:50.500501 6090 hybrid_clock.cc:648] HybridClock initialized: now 1790280830500461 us; error 55 us; skew 500 ppm
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:50.503528 6090 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:13:50.504416 6090 webserver.cc:460] Webserver started at http://127.29.83.131:37837/ using document root <none> and password file <none>
I20260924 20:13:50.505020 6090 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:50.505124 6090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:50.505359 6090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:13:50.507444 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "d55a025fa36844dfaada1673d3c971b0"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "9ad8b5f30205eabe0a2f18bfe5aa2cfe"
server_key_iv: "d28ea94ef13b60ea3f05845e5ad6512d"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.507999 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "d55a025fa36844dfaada1673d3c971b0"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "9ad8b5f30205eabe0a2f18bfe5aa2cfe"
server_key_iv: "d28ea94ef13b60ea3f05845e5ad6512d"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.511799 6090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 20:13:50.514526 6105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.515870 6090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:50.516004 6090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "d55a025fa36844dfaada1673d3c971b0"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "9ad8b5f30205eabe0a2f18bfe5aa2cfe"
server_key_iv: "d28ea94ef13b60ea3f05845e5ad6512d"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.516155 6090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:50.529179 6090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:50.532070 6090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:50.532281 6090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:50.532867 6090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:50.533679 6090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:50.533751 6090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.533862 6090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:50.533916 6090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.543089 6090 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:38969
I20260924 20:13:50.543111 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:38969 every 8 connection(s)
I20260924 20:13:50.544024 6090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:50.544250 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6090
I20260924 20:13:50.544384 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:13:50.544751 30030 external_mini_cluster.cc:1428] Setting key b0f29fd9282fc09420053295cf8006d4
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:13:50.558365 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.546250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:33841 (local address 127.29.83.190:43721)
0924 20:13:50.546397 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:50.546401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:50.547347 (+ 946us) server_negotiation.cc:407] Connection header received
0924 20:13:50.548612 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:50.548614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:50.548659 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:50.548727 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:50.549894 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.550436 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.551669 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.551888 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.551968 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.554497 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:50.554500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:50.554503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:50.554529 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:50.555978 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.556584 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.556586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.556587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.556667 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:50.557067 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:50.557070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:50.557071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:50.557253 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:50.557340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:50.558032 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 20:13:50.558181 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:13:50.559295 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:50.546569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:33841)
0924 20:13:50.547161 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:50.547239 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:50.548435 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:50.548883 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:50.548889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:50.549268 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:50.549730 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.549742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.550666 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.550669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.551550 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:50.551560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:50.552025 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:50.552028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:50.552174 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:50.552830 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:50.552848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:50.554321 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:13:50.556142 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.556146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.556158 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.556497 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.556787 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:13:50.556790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:13:50.556791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:13:50.556936 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:13:50.557383 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:13:50.557393 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:13:50.557706 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 20:13:50.557947 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 20:13:50.558317 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260924 20:13:50.560389 6219 heartbeater.cc:347] Connected to a master server at 127.29.83.190:43721
I20260924 20:13:50.560635 6219 heartbeater.cc:464] Registering TS with master...
I20260924 20:13:50.561113 6219 heartbeater.cc:511] Master 127.29.83.190:43721 requested a full tablet report, sending...
I20260924 20:13:50.562124 5760 ts_manager.cc:194] Registered new tserver with Master: d55a025fa36844dfaada1673d3c971b0 (127.29.83.131:38969)
I20260924 20:13:50.562639 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:33841
I20260924 20:13:50.571118 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
Entry for principal oryx/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab.
WARNING: no policy specified for oryx/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
Entry for principal oryx/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab.
WARNING: no policy specified for oryx/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
Entry for principal oryx/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab.
I20260924 20:13:50.646535 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5822
I20260924 20:13:50.652180 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5956
I20260924 20:13:50.657416 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6090
I20260924 20:13:50.662683 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 5728
I20260924 20:13:50.669112 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:43721
--webserver_interface=127.29.83.190
--webserver_port=42845
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:43721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:13:50.781342 6228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:50.781623 6228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:50.781664 6228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:50.785627 6228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:13:50.785688 6228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:50.785710 6228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:50.785727 6228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:13:50.785742 6228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:13:50.791121 6228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:43721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:50.792084 6228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:50.793035 6228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:50.801313 6233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.801378 6234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.801354 6228 server_base.cc:1034] running on GCE node
W20260924 20:13:50.801532 6236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.802198 6228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:50.803292 6228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:50.804545 6228 hybrid_clock.cc:648] HybridClock initialized: now 1790280830804517 us; error 49 us; skew 500 ppm
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:50.807602 6228 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:13:50.808526 6228 webserver.cc:460] Webserver started at http://127.29.83.190:42845/ using document root <none> and password file <none>
I20260924 20:13:50.809039 6228 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:50.809113 6228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:50.813112 6228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:13:50.815351 6243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:50.816625 6228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:13:50.816772 6228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "72da6c5188344f67a7f8d97338084bb3"
format_stamp: "Formatted at 2026-09-24 20:13:49 on dist-test-slave-52fr"
server_key: "262f9192e22ff2913e1b2879f0af975a"
server_key_iv: "48cc0a1d706b7823547e8ad83615a550"
server_key_version: "encryptionkey@0"
I20260924 20:13:50.817317 6228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:50.836644 6228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:50.840236 6228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:50.840449 6228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:50.848361 6228 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:43721
I20260924 20:13:50.848374 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:43721 every 8 connection(s)
I20260924 20:13:50.849514 6228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:13:50.853822 6296 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:13:50.855491 6296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Bootstrap starting.
I20260924 20:13:50.855995 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6228
I20260924 20:13:50.856475 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:41663
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:43721
--webserver_port=32829
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:13:50.873301 6296 log.cc:824] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:13:50.879979 6296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:13:50.880398 6296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Bootstrap complete.
I20260924 20:13:50.884686 6296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:50.885080 6296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da6c5188344f67a7f8d97338084bb3, State: Initialized, Role: FOLLOWER
I20260924 20:13:50.885599 6296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:50.885716 6296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:13:50.885824 6296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:13:50.885946 6296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:13:50.887923 6296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:50.888262 6296 leader_election.cc:302] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72da6c5188344f67a7f8d97338084bb3; no voters:
I20260924 20:13:50.888602 6296 leader_election.cc:288] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:13:50.888697 6300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:13:50.888883 6300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [term 2 LEADER]: Becoming Leader. State: Replica: 72da6c5188344f67a7f8d97338084bb3, State: Running, Role: LEADER
I20260924 20:13:50.889204 6300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } }
I20260924 20:13:50.889879 6296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:13:50.890151 6302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72da6c5188344f67a7f8d97338084bb3. Latest consensus state: current_term: 2 leader_uuid: "72da6c5188344f67a7f8d97338084bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } } }
I20260924 20:13:50.890275 6302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:50.890522 6301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72da6c5188344f67a7f8d97338084bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da6c5188344f67a7f8d97338084bb3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 43721 } } }
I20260924 20:13:50.890621 6301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:13:50.890733 6304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:13:50.895218 6304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:13:50.895859 6304 catalog_manager.cc:1259] Loaded cluster ID: 9fdfcbb3be95402f85bd00966b9ce45f
I20260924 20:13:50.895923 6304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:13:50.898286 6304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:13:50.898952 6304 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72da6c5188344f67a7f8d97338084bb3: Loaded TSK: 0
I20260924 20:13:50.900198 6304 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:13:50.972864 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:50.973161 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:50.973233 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:50.977308 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:50.977404 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:50.977499 6298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:13:50.982966 6298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.129.keytab
--principal=oryx/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:41663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=32829
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:50.984088 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:50.985126 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:50.994911 6323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:50.994923 6322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.995183 6298 server_base.cc:1034] running on GCE node
W20260924 20:13:50.995370 6325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:50.995885 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:50.996676 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:50.997836 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790280830997805 us; error 40 us; skew 500 ppm
Sep 24 20:13:50 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280830, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:51.001219 6298 init.cc:377] Logged in from keytab as oryx/127.29.83.129@KRBTEST.COM (short username oryx)
I20260924 20:13:51.002180 6298 webserver.cc:460] Webserver started at http://127.29.83.129:32829/ using document root <none> and password file <none>
I20260924 20:13:51.002734 6298 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:51.002806 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:51.006660 6298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:51.008860 6332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.010248 6298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:51.010382 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "4dd1c4afd5b443be8615787b971b1b91"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "c59dccfd1f1db84b79281a424b600df0"
server_key_iv: "d79d668a0b640707614ad832acf3305a"
server_key_version: "encryptionkey@0"
I20260924 20:13:51.010928 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:51.031659 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:51.034956 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:51.035164 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:51.035792 6298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:51.036617 6298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:51.036688 6298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.036760 6298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:51.036811 6298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.045887 6298 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:41663
I20260924 20:13:51.045914 6445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:41663 every 8 connection(s)
I20260924 20:13:51.046869 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:13:51.053395 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6298
I20260924 20:13:51.053902 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:45201
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:43721
--webserver_port=35965
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:13:51.060230 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.048953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:53213 (local address 127.29.83.190:43721)
0924 20:13:51.049380 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.049399 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.049717 (+ 318us) server_negotiation.cc:407] Connection header received
0924 20:13:51.051250 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.051263 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.051547 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.051837 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.052885 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.054026 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.055193 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.055668 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.055842 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.059485 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:53213: BlockingRecv error: recv got EOF from 127.29.83.129:53213 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":131,"threads_started":1}
I20260924 20:13:51.060235 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.049025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:53213)
0924 20:13:51.049506 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.049566 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.050857 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.051927 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.051933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.052281 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.052732 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.052745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.054199 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.054202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.055053 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.055062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.055833 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.055837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.056033 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.056655 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.056671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.059283 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":153,"threads_started":1}
W20260924 20:13:51.060988 6446 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:43721 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.83.190:43721: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.066560 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.061476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:39407)
0924 20:13:51.061592 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.061607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.061682 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.061970 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.061972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.062145 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.062232 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.062237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.062954 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.062959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.063498 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.063506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.063955 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.063958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.064053 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.064551 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.064562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.066370 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:13:51.066596 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.061581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:39407 (local address 127.29.83.190:43721)
0924 20:13:51.061728 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.061734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.061748 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:51.061816 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.061821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.061873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.061955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.062344 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.062851 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.063624 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.063840 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.063942 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.066494 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:39407: BlockingRecv error: recv got EOF from 127.29.83.129:39407 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.072835 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.067429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:35627)
0924 20:13:51.067562 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.067577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.067652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.067875 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.067877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.068034 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.068130 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.068136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.068825 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.068828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.069728 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.069739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.070217 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.070223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.070334 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.070829 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.070843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.072695 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 20:13:51.072949 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.067507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:35627 (local address 127.29.83.190:43721)
0924 20:13:51.067612 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.067622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.067636 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:51.067737 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.067739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.067775 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.067840 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.068252 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.068728 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.069877 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.070085 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.070183 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.072821 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:35627: BlockingRecv error: recv got EOF from 127.29.83.129:35627 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 20:13:51.073091 6446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:13:51.170831 6450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:51.171085 6450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:51.171129 6450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:51.175081 6450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:51.175140 6450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:51.175197 6450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:13:51.180420 6450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.130.keytab
--principal=oryx/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:45201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=35965
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:51.181397 6450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:51.182379 6450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:51.191702 6456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:51.191730 6455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:51.191890 6458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:51.192144 6450 server_base.cc:1034] running on GCE node
I20260924 20:13:51.192538 6450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:51.193209 6450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:51.194361 6450 hybrid_clock.cc:648] HybridClock initialized: now 1790280831194341 us; error 31 us; skew 500 ppm
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:51.197062 6450 init.cc:377] Logged in from keytab as oryx/127.29.83.130@KRBTEST.COM (short username oryx)
I20260924 20:13:51.198113 6450 webserver.cc:460] Webserver started at http://127.29.83.130:35965/ using document root <none> and password file <none>
I20260924 20:13:51.198691 6450 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:51.198796 6450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:51.202721 6450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:13:51.204807 6465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.206176 6450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:13:51.206312 6450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "2fbb04ea9ae14b86a4c9654b26f0ae98"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "428cf85ffab9f8d033aa1c6a75307649"
server_key_iv: "db2c1089bcfa4cd31fa86b7136d55dc1"
server_key_version: "encryptionkey@0"
I20260924 20:13:51.206868 6450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:51.232975 6450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:51.236204 6450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:51.236426 6450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:51.237051 6450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:51.237926 6450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:51.237998 6450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.238068 6450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:51.238116 6450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.247308 6450 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:45201
I20260924 20:13:51.247330 6578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:45201 every 8 connection(s)
I20260924 20:13:51.248299 6450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:13:51.250646 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6450
I20260924 20:13:51.251077 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:38969
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:43721
--webserver_port=37837
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:13:51.260156 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.250245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:44081 (local address 127.29.83.190:43721)
0924 20:13:51.250435 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.250440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.251280 (+ 840us) server_negotiation.cc:407] Connection header received
0924 20:13:51.252729 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.252732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.252794 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.252897 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.254147 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.254797 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.255983 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.256220 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.256336 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.260049 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:44081: BlockingRecv error: recv got EOF from 127.29.83.130:44081 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:13:51.261092 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.250631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:44081)
0924 20:13:51.251129 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.251182 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.252514 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.253077 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.253085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.253432 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.253996 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.254012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.254968 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.254971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.255854 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.255863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.256378 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.256382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.256553 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.257185 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.257201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.259935 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
W20260924 20:13:51.261595 6579 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:43721 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.83.190:43721: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.267545 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.262057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:57793)
0924 20:13:51.262207 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.262222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.262333 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.262662 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.262665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.262834 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.262943 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.262948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.263626 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.263629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.264553 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.264563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.264984 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.264988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.265085 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.265556 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.265570 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.267421 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 20:13:51.267692 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.262168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:57793 (local address 127.29.83.190:43721)
0924 20:13:51.262290 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.262293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.262309 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:13:51.262514 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.262516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.262555 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.262645 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.263046 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.263527 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.264677 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.264880 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.264960 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.267565 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:57793: BlockingRecv error: recv got EOF from 127.29.83.130:57793 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.273906 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.268476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:46415)
0924 20:13:51.268671 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.268685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.268751 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.268958 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.268960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.269075 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.269156 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.269160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.270178 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.270181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.270693 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.270700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.271291 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.271294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.271361 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.271912 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.271926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.273785 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260924 20:13:51.274170 6579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:13:51.274416 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.268587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:46415 (local address 127.29.83.190:43721)
0924 20:13:51.268741 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.268745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.268759 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:51.268815 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.268818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.268859 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.268938 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.269263 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.270071 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.270804 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.271144 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.271252 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.274316 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:46415: BlockingRecv error: recv got EOF from 127.29.83.130:46415 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 20:13:51.369266 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:13:51.369524 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:13:51.369575 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:13:51.373639 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:13:51.373708 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:13:51.373828 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:13:51.379354 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/krb5kdc/oryx_127.29.83.131.keytab
--principal=oryx/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:38969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=37837
--tserver_master_addrs=127.29.83.190:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:13:51.380458 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:13:51.381546 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:13:51.391788 6588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:51.391788 6590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:13:51.391829 6587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:13:51.392112 6582 server_base.cc:1034] running on GCE node
I20260924 20:13:51.392690 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:13:51.393297 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:13:51.394491 6582 hybrid_clock.cc:648] HybridClock initialized: now 1790280831394456 us; error 47 us; skew 500 ppm
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:13:51.397394 6582 init.cc:377] Logged in from keytab as oryx/127.29.83.131@KRBTEST.COM (short username oryx)
I20260924 20:13:51.398231 6582 webserver.cc:460] Webserver started at http://127.29.83.131:37837/ using document root <none> and password file <none>
I20260924 20:13:51.398809 6582 fs_manager.cc:359] Metadata directory not provided
I20260924 20:13:51.398880 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:13:51.402760 6582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:13:51.405057 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.406456 6582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:13:51.406605 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "d55a025fa36844dfaada1673d3c971b0"
format_stamp: "Formatted at 2026-09-24 20:13:50 on dist-test-slave-52fr"
server_key: "9ad8b5f30205eabe0a2f18bfe5aa2cfe"
server_key_iv: "d28ea94ef13b60ea3f05845e5ad6512d"
server_key_version: "encryptionkey@0"
I20260924 20:13:51.407217 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:13:51.425155 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:13:51.428529 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:13:51.428750 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:13:51.429378 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:13:51.430248 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:13:51.430296 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.430336 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:13:51.430392 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:13:51.439661 6582 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:38969
I20260924 20:13:51.439682 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:38969 every 8 connection(s)
I20260924 20:13:51.440712 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:13:51.447618 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6582
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:13:51.453320 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.442624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:34167 (local address 127.29.83.190:43721)
0924 20:13:51.442800 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.442805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.443735 (+ 930us) server_negotiation.cc:407] Connection header received
0924 20:13:51.445098 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.445101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.445150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.445228 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.446511 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.447060 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.448459 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.448800 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.448920 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.453198 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:34167: BlockingRecv error: recv got EOF from 127.29.83.131:34167 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:13:51.454133 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.443033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:34167)
0924 20:13:51.443546 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.443595 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.444858 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.445415 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.445422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.445837 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.446348 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.446362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.447203 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.447206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.448311 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.448329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.448978 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.448983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.449246 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.450020 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.450036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.453092 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":105,"threads_started":1}
W20260924 20:13:51.454720 6711 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:43721 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.83.190:43721: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
I20260924 20:13:51.458773 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.448728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.29.83.190:43721)
0924 20:13:51.449025 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.449043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.449057 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:51.449092 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.449095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.449141 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.449247 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.450165 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.450931 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.452120 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.452453 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.452578 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.454080 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:13:51.454087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:13:51.454106 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:13:51.454142 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:13:51.456533 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:51.456979 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:51.456993 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:51.456998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:51.457119 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:13:51.457409 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:13:51.457414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:13:51.457416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:13:51.457851 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:13:51.458054 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 20:13:51.458375 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 20:13:51.458644 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.461153 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.455227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:37249)
0924 20:13:51.455356 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.455368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.455452 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.455798 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.455800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.455933 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.456021 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.456026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.456937 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.456942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.457509 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.457516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.458041 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.458045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.458145 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.458622 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.458643 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.461057 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 20:13:51.461345 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.455403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37249 (local address 127.29.83.190:43721)
0924 20:13:51.455573 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.455578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.455592 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:13:51.455629 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.455633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.455683 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.455766 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.456134 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.456833 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.457634 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.457925 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.458028 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.461223 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:37249: BlockingRecv error: recv got EOF from 127.29.83.131:37249 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:51 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:51.467039 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:51.461878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:56989)
0924 20:13:51.461990 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:51.462006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:51.462081 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:51.462392 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:51.462394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:51.462532 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:51.462608 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.462626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.463423 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.463426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.463984 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:51.463992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.464462 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.464465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:51.464541 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.465080 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:51.465098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:51.466923 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:13:51.467252 6448 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:51.461950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:56989 (local address 127.29.83.190:43721)
0924 20:13:51.462071 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:51.462078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:51.462093 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:51.462198 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:51.462200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:51.462237 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:51.462301 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:51.462816 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.463316 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.464109 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:51.464320 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:51.464423 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:51.467089 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:56989: BlockingRecv error: recv got EOF from 127.29.83.131:56989 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:13:51.467340 6711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:52.079156 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:52.073679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:51443)
0924 20:13:52.074011 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:52.074024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:52.074124 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:52.074366 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:52.074368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:52.074552 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:52.074644 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.074649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.075417 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.075421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.076090 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.076099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.076538 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.076541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.076613 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.077153 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:52.077165 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:52.079034 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
W20260924 20:13:52.079339 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:52.073794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:51443 (local address 127.29.83.190:43721)
0924 20:13:52.074037 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:52.074041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:52.074079 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:13:52.074209 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:52.074211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:52.074254 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:52.074326 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:52.074755 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.075293 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.076205 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.076428 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.076506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.079195 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:51443: BlockingRecv error: recv got EOF from 127.29.83.129:51443 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":52,"threads_started":1}
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:52.279995 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:52.274832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:35871)
0924 20:13:52.275082 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:52.275095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:52.275194 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:52.275407 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:52.275409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:52.275573 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:52.275671 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.275676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.276397 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.276399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.276974 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.276982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.277429 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.277431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.277491 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.278106 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:52.278128 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:52.279872 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
W20260924 20:13:52.280135 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:52.274883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:35871 (local address 127.29.83.190:43721)
0924 20:13:52.275042 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:52.275047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:52.275152 (+ 105us) server_negotiation.cc:407] Connection header received
0924 20:13:52.275260 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:52.275262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:52.275305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:52.275373 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:52.275776 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.276306 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.277089 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.277331 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.277413 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.280016 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:35871: BlockingRecv error: recv got EOF from 127.29.83.130:35871 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:52 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:52.474272 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:52.468197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:46077)
0924 20:13:52.468476 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:52.468490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:52.468592 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:52.468876 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:52.468879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:52.469056 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:52.469156 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.469161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.470266 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.470271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.471010 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:52.471020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.471499 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.471502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:52.471587 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.472164 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:52.472176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:52.474091 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
W20260924 20:13:52.474357 6717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:52.468361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:46077 (local address 127.29.83.190:43721)
0924 20:13:52.468493 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:52.468496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:52.468561 (+ 65us) server_negotiation.cc:407] Connection header received
0924 20:13:52.468675 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:52.468678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:52.468724 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:52.468807 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:52.469321 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.470157 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.471135 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:52.471377 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:52.471484 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:52.474236 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:46077: BlockingRecv error: recv got EOF from 127.29.83.131:46077 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:53.087821 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:53.080106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:53201)
0924 20:13:53.080417 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:53.080437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:53.080554 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:53.080920 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:53.080923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:53.081119 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:53.081239 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.081245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.082603 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.082605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.083191 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.083201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.083955 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.083959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.084052 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.085005 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:53.085027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:53.087571 (+ 2544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
W20260924 20:13:53.087853 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:53.080231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:53201 (local address 127.29.83.190:43721)
0924 20:13:53.080592 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:53.080599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:53.080620 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:13:53.080675 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:53.080681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:53.080747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:53.080863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:53.081381 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.082439 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.083314 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.083791 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.083927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.087738 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:53201: BlockingRecv error: recv got EOF from 127.29.83.129:53201 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:53.286859 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:53.280883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:54991)
0924 20:13:53.281145 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:53.281158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:53.281248 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:53.281617 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:53.281620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:53.281830 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:53.281925 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.281931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.282793 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.282796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.283364 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.283371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.283910 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.283913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.283985 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.284510 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:53.284521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:53.286664 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
W20260924 20:13:53.286918 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:53.281027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:54991 (local address 127.29.83.190:43721)
0924 20:13:53.281195 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:53.281198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:53.281213 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:13:53.281414 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:53.281416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:53.281461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:53.281563 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:53.282073 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.282640 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.283519 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.283772 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.283873 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.286808 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:54991: BlockingRecv error: recv got EOF from 127.29.83.130:54991 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:53 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:53.482069 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:53.475185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:39893)
0924 20:13:53.475721 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:53.475734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:53.475831 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:53.476125 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:53.476127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:53.476310 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:53.476408 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.476414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.477502 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.477505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.478142 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:53.478152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.478700 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.478702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:53.478888 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.479783 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:53.479805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:53.481917 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":218,"threads_started":1}
W20260924 20:13:53.482193 6721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:53.475580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:39893 (local address 127.29.83.190:43721)
0924 20:13:53.475771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:53.475775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:53.475792 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:13:53.475923 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:53.475925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:53.475971 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:53.476058 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:53.476590 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.477293 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.478277 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:53.478562 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:53.478826 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:53.482057 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:39893: BlockingRecv error: recv got EOF from 127.29.83.131:39893 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:54.095517 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:54.088909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:53253)
0924 20:13:54.089261 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:54.089298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:54.089407 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:54.089791 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:54.089795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:54.090078 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:54.090229 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.090240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.091035 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.091038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.091741 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.091751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.092270 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.092274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.092345 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.093210 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:54.093223 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:54.095391 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
W20260924 20:13:54.095799 6725 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:54.088962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:53253 (local address 127.29.83.190:43721)
0924 20:13:54.089343 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:54.089348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:54.089370 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:13:54.089528 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:54.089533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:54.089590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:54.089687 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:54.090349 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.090922 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.091874 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.092139 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.092227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.095596 (+ 3369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:53253: BlockingRecv error: recv got EOF from 127.29.83.129:53253 (error 108)
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":93,"threads_started":1}
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:54.293581 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:54.287632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:42781)
0924 20:13:54.287954 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:54.287970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:54.288116 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:54.288389 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:54.288392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:54.288606 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:54.288696 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.288701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.289487 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.289494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.290116 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.290125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.290600 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.290603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.290690 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.291264 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:54.291277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:54.293414 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":84,"threads_started":1}
W20260924 20:13:54.293661 6725 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:54.287767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:42781 (local address 127.29.83.190:43721)
0924 20:13:54.287951 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:54.287956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:54.288055 (+ 99us) server_negotiation.cc:407] Connection header received
0924 20:13:54.288226 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:54.288229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:54.288281 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:54.288368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:54.288814 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.289358 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.290244 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.290473 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.290576 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.293566 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:42781: BlockingRecv error: recv got EOF from 127.29.83.130:42781 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:54 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:54.489729 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:54.483137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:47225)
0924 20:13:54.483479 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:54.483493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:54.483595 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:54.483922 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:54.483928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:54.484141 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:54.484236 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.484242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.485070 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.485073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.485644 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:54.485652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.486176 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.486179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:54.486252 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.486934 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:54.486948 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:54.489512 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":131,"threads_started":1}
W20260924 20:13:54.489884 6725 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:54.483194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:47225 (local address 127.29.83.190:43721)
0924 20:13:54.483352 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:54.483357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:54.483581 (+ 224us) server_negotiation.cc:407] Connection header received
0924 20:13:54.483692 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:54.483696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:54.483759 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:54.483839 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:54.484331 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.484942 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.485786 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:54.486034 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:54.486119 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:54.489714 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:47225: BlockingRecv error: recv got EOF from 127.29.83.131:47225 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:55.105535 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:55.096292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:58373)
0924 20:13:55.096542 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:55.096568 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:55.096684 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:55.096927 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:55.096929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:55.097103 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:55.097201 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.097206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.098077 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.098080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.098665 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.098675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.099070 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.099072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.099131 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.103422 (+ 4291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:55.103443 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:55.105401 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
W20260924 20:13:55.105683 6729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:55.096389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:58373 (local address 127.29.83.190:43721)
0924 20:13:55.096657 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:55.096661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:55.096673 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:55.096769 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:55.096771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:55.096815 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:55.096892 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:55.097301 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.097975 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.098771 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.098978 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.099049 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.105545 (+ 6496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:58373: BlockingRecv error: recv got EOF from 127.29.83.129:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:55.299930 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:55.294543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:36353)
0924 20:13:55.294870 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:55.294883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:55.294992 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:55.295232 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:55.295234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:55.295409 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:55.295511 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.295516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.296295 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.296298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.296875 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.296884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.297353 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.297356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.297426 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.298009 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:55.298020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:55.299816 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
W20260924 20:13:55.300068 6729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:55.294593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:36353 (local address 127.29.83.190:43721)
0924 20:13:55.294763 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:55.294769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:55.294955 (+ 186us) server_negotiation.cc:407] Connection header received
0924 20:13:55.295071 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:55.295074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:55.295124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:55.295201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:55.295636 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.296168 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.297006 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.297230 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.297308 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.299950 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:36353: BlockingRecv error: recv got EOF from 127.29.83.130:36353 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:55 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:55.496930 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:55.490723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:36765)
0924 20:13:55.491068 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:55.491082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:55.491207 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:55.491513 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:55.491515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:55.491784 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:55.491888 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.491895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.492765 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.492768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.493347 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:55.493355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.493964 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.493967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:55.494041 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.494563 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:55.494575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:55.496784 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
W20260924 20:13:55.497117 6729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:55.490808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:36765 (local address 127.29.83.190:43721)
0924 20:13:55.490959 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:55.490972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:55.491183 (+ 211us) server_negotiation.cc:407] Connection header received
0924 20:13:55.491296 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:55.491298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:55.491349 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:55.491437 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:55.492086 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.492640 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.493493 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:55.493747 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:55.493916 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:55.496944 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:36765: BlockingRecv error: recv got EOF from 127.29.83.131:36765 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:56.113385 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:56.106714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:59677)
0924 20:13:56.107117 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:56.107133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:56.107240 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:56.107651 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:56.107654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:56.107837 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:56.107933 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.107938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.108764 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.108767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.109413 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.109423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.110010 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.110013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.110088 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.110913 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:56.110926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:56.113252 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
W20260924 20:13:56.113554 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:56.106905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:59677 (local address 127.29.83.190:43721)
0924 20:13:56.107181 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:56.107185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:56.107201 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:13:56.107382 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:56.107385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:56.107433 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:56.107555 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:56.108082 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.108637 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.109603 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.109879 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.109967 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.113408 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:59677: BlockingRecv error: recv got EOF from 127.29.83.129:59677 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:56.306581 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:56.300909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:44615)
0924 20:13:56.301280 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:56.301297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:56.301431 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:56.301724 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:56.301727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:56.302007 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:56.302113 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.302119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.303029 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.303032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.303600 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.303607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.304045 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.304047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.304107 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.304588 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:56.304600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:56.306461 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":184,"threads_started":1}
W20260924 20:13:56.306794 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:56.301121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:44615 (local address 127.29.83.190:43721)
0924 20:13:56.301287 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:56.301290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:56.301393 (+ 103us) server_negotiation.cc:407] Connection header received
0924 20:13:56.301548 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:56.301550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:56.301594 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:56.301696 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:56.302214 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.302915 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.303717 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.303939 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.304020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.306641 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:44615: BlockingRecv error: recv got EOF from 127.29.83.130:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:56 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:56.503245 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:56.497833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:37661)
0924 20:13:56.498110 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:56.498131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:56.498230 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:56.498497 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:56.498499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:56.498679 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:56.498767 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.498772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.499530 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.499533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.500108 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:56.500115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.500642 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.500645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:56.500713 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.501229 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:56.501241 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:56.503127 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
W20260924 20:13:56.503386 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:56.497912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37661 (local address 127.29.83.190:43721)
0924 20:13:56.498077 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:56.498082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:56.498209 (+ 127us) server_negotiation.cc:407] Connection header received
0924 20:13:56.498317 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:56.498319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:56.498364 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:56.498433 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:56.498900 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.499420 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.500245 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:56.500506 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:56.500604 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:56.503265 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:37661: BlockingRecv error: recv got EOF from 127.29.83.131:37661 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:57.120059 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:57.114306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:36467)
0924 20:13:57.114584 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:57.114598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:57.114720 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:57.114970 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:57.114972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:57.115163 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:57.115320 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.115327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.116235 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.116238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.116814 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.116824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.117289 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.117292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.117368 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.117963 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:57.117977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:57.119866 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260924 20:13:57.120121 6737 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:57.114369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:36467 (local address 127.29.83.190:43721)
0924 20:13:57.114715 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:57.114720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:57.114742 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:13:57.114784 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:57.114786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:57.114828 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:57.114906 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:57.115461 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.116101 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.116934 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.117165 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.117247 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.119992 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:36467: BlockingRecv error: recv got EOF from 127.29.83.129:36467 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:57.313990 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:57.307442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:60143)
0924 20:13:57.307740 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:57.307756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:57.307917 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:57.308283 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:57.308288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:57.308530 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:57.308674 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.308683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.309851 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.309855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.310620 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.310651 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.311144 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.311147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.311212 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.311723 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:57.311734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:57.313815 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":124,"threads_started":1}
W20260924 20:13:57.314069 6737 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:57.307552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:60143 (local address 127.29.83.190:43721)
0924 20:13:57.307832 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:57.307838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:57.307861 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:13:57.308042 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:57.308049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:57.308123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:57.308233 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:57.308815 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.309693 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.310813 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.311043 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.311128 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.313956 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:60143: BlockingRecv error: recv got EOF from 127.29.83.130:60143 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:57 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:57.510161 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:57.504086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:39281)
0924 20:13:57.504427 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:57.504448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:57.504586 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:57.504869 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:57.504872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:57.505096 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:57.505186 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.505192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.506052 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.506057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.506624 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:57.506633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.507251 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.507255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:57.507367 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.507972 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:57.507985 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:57.509971 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
W20260924 20:13:57.510465 6737 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:57.504217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:39281 (local address 127.29.83.190:43721)
0924 20:13:57.504391 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:57.504396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:57.504552 (+ 156us) server_negotiation.cc:407] Connection header received
0924 20:13:57.504696 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:57.504700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:57.504758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:57.504846 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:57.505314 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.505927 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.506753 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:57.506992 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:57.507253 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:57.510184 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:39281: BlockingRecv error: recv got EOF from 127.29.83.131:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:58.127136 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:58.121055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:38709)
0924 20:13:58.121459 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:58.121477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:58.121601 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:58.121979 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:58.121982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:58.122224 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:58.122334 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.122340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.123152 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.123158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.123750 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.123759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.124246 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.124250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.124335 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.124913 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:58.124925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:58.126954 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":82,"threads_started":1}
W20260924 20:13:58.127260 6741 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:58.121119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:38709 (local address 127.29.83.190:43721)
0924 20:13:58.121389 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:58.121392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:58.121591 (+ 199us) server_negotiation.cc:407] Connection header received
0924 20:13:58.121738 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:58.121744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:58.121835 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:58.121954 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:58.122462 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.123018 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.123881 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.124130 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.124242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.127113 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:38709: BlockingRecv error: recv got EOF from 127.29.83.129:38709 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:58.320477 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:58.315212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:41793)
0924 20:13:58.315484 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:58.315501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:58.315606 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:58.315871 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:58.315874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:58.316077 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:58.316174 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.316180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.316932 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.316935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.317499 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.317507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.318009 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.318012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.318091 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.318587 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:58.318600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:58.320348 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
W20260924 20:13:58.320628 6741 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:58.315237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:41793 (local address 127.29.83.190:43721)
0924 20:13:58.315376 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:58.315380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:58.315585 (+ 205us) server_negotiation.cc:407] Connection header received
0924 20:13:58.315676 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:58.315678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:58.315723 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:58.315792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:58.316278 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.316820 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.317609 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.317839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.317951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.320497 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:41793: BlockingRecv error: recv got EOF from 127.29.83.130:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:58 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:58.517035 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:58.511238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:40977)
0924 20:13:58.511758 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:58.511775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:58.511911 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:58.512219 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:58.512222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:58.512438 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:58.512529 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.512535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.513345 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.513348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.513986 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:58.513995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.514449 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.514452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:58.514512 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.515051 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:58.515064 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:58.516916 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":77,"threads_started":1}
W20260924 20:13:58.517160 6741 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:58.511348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:40977 (local address 127.29.83.190:43721)
0924 20:13:58.511547 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:58.511552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:58.511921 (+ 369us) server_negotiation.cc:407] Connection header received
0924 20:13:58.512020 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:58.512026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:58.512097 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:58.512193 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:58.512699 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.513253 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.514114 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:58.514353 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:58.514433 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:58.517036 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:40977: BlockingRecv error: recv got EOF from 127.29.83.131:40977 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:59.133558 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:59.128048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:57837)
0924 20:13:59.128435 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:59.128474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:59.128604 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:59.128836 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:59.128838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:59.129012 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:59.129108 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.129113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.129889 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.129891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.130457 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.130465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.130914 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.130917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.130982 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.131499 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:59.131510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:59.133423 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
W20260924 20:13:59.133770 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:59.128229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:57837 (local address 127.29.83.190:43721)
0924 20:13:59.128538 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:59.128542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:59.128554 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:13:59.128688 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:59.128690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:59.128733 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:59.128801 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:59.129213 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.129796 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.130565 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.130808 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.130888 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.133605 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:57837: BlockingRecv error: recv got EOF from 127.29.83.129:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:59.326916 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:59.321266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:53245)
0924 20:13:59.321554 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:59.321568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:59.321665 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:59.321913 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:59.321915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:59.322099 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:59.322196 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.322201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.323083 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.323086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.323666 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.323675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.324174 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.324177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.324244 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.324735 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:59.324747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:59.326739 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":129,"threads_started":1}
W20260924 20:13:59.326982 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:59.321354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:53245 (local address 127.29.83.190:43721)
0924 20:13:59.321519 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:59.321524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:59.321638 (+ 114us) server_negotiation.cc:407] Connection header received
0924 20:13:59.321742 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:59.321744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:59.321810 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:59.321878 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:59.322295 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.322961 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.323813 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.324060 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.324151 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.326874 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:53245: BlockingRecv error: recv got EOF from 127.29.83.130:53245 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:13:59 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:13:59.524647 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:13:59.518100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:41845)
0924 20:13:59.518424 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 20:13:59.518437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:13:59.518574 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:13:59.518964 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:13:59.518967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:13:59.519155 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:13:59.519256 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.519262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.520449 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.520454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.521320 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:13:59.521332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.521964 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.521968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:13:59.522045 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.522681 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:13:59.522695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:13:59.524518 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
W20260924 20:13:59.524891 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:13:59.518189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:41845 (local address 127.29.83.190:43721)
0924 20:13:59.518372 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 20:13:59.518376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:13:59.518610 (+ 234us) server_negotiation.cc:407] Connection header received
0924 20:13:59.518666 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:13:59.518668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:13:59.518721 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:13:59.518816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:13:59.519419 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.520287 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.521471 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:13:59.521842 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:13:59.521941 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:13:59.524764 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:41845: BlockingRecv error: recv got EOF from 127.29.83.131:41845 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:00.144001 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:00.134604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:52369)
0924 20:14:00.134933 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:00.134947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:00.135043 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:00.135362 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:00.135365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:00.135555 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:00.135658 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.135664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.136495 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.136498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.137091 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.137101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.137556 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.137592 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.137680 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.141897 (+ 4217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:00.141910 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:00.143885 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
W20260924 20:14:00.144177 6749 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:00.134718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:52369 (local address 127.29.83.190:43721)
0924 20:14:00.134971 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:00.134975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:00.134993 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:14:00.135142 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:00.135145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:00.135190 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:00.135277 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:00.135804 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.136371 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.137201 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.137430 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.137505 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.144039 (+ 6534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:52369: BlockingRecv error: recv got EOF from 127.29.83.129:52369 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:00.334662 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:00.327909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:38541)
0924 20:14:00.328218 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:00.328232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:00.328338 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:00.328707 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:00.328709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:00.328910 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:00.329014 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.329022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.329963 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.329966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.330856 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.330866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.331390 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.331393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.331492 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.332027 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:00.332040 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:00.334524 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
W20260924 20:14:00.335091 6749 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:00.328093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:38541 (local address 127.29.83.190:43721)
0924 20:14:00.328303 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:00.328307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:00.328324 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:14:00.328444 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:00.328447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:00.328533 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:00.328613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:00.329166 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.329784 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.330999 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.331261 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.331392 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.334966 (+ 3574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:38541: BlockingRecv error: recv got EOF from 127.29.83.130:38541 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:00 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:00.531438 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:00.525569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:33125)
0924 20:14:00.525923 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:00.525937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:00.526072 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:00.526456 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:00.526459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:00.526703 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:00.526798 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.526804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.527579 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.527583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.528150 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:00.528158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.528624 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.528628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:00.528712 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.529262 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:00.529274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:00.531261 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
W20260924 20:14:00.531499 6749 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:00.525752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:33125 (local address 127.29.83.190:43721)
0924 20:14:00.525912 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:00.525916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:00.526026 (+ 110us) server_negotiation.cc:407] Connection header received
0924 20:14:00.526229 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:00.526232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:00.526290 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:00.526435 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:00.526920 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.527456 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.528272 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:00.528508 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:00.528623 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:00.531390 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:33125: BlockingRecv error: recv got EOF from 127.29.83.131:33125 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:01.151587 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:01.145095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:57777)
0924 20:14:01.145450 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:01.145485 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:01.145616 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:01.146026 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:01.146029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:01.146239 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:01.146339 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.146345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.147235 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.147238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.147865 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.147875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.148380 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.148382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.148451 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.149010 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:01.149023 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:01.151405 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":83,"threads_started":1}
W20260924 20:14:01.151707 6753 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:01.145229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:57777 (local address 127.29.83.190:43721)
0924 20:14:01.145471 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:01.145476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:01.145740 (+ 264us) server_negotiation.cc:407] Connection header received
0924 20:14:01.145812 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:01.145816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:01.145874 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:01.145951 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:01.146511 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.147101 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.147999 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.148245 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.148347 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.151568 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:57777: BlockingRecv error: recv got EOF from 127.29.83.129:57777 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:01.342487 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:01.335562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:35645)
0924 20:14:01.335870 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:01.335888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:01.336006 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:01.336288 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:01.336294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:01.336537 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:01.336679 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.336689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.337859 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.337863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.338815 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.338826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.339480 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.339484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.339574 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.340212 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:01.340227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:01.342317 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
W20260924 20:14:01.342775 6753 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:01.335606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:35645 (local address 127.29.83.190:43721)
0924 20:14:01.335765 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:01.335771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:01.335962 (+ 191us) server_negotiation.cc:407] Connection header received
0924 20:14:01.336112 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:01.336114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:01.336161 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:01.336258 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:01.336832 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.337704 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.338947 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.339343 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.339439 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.342439 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:35645: BlockingRecv error: recv got EOF from 127.29.83.130:35645 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:01 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:01.537902 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:01.532327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:37083)
0924 20:14:01.532618 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:01.532631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:01.532720 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:01.532950 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:01.532952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:01.533130 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:01.533225 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.533230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.534088 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.534093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.534657 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:01.534665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.535131 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.535134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:01.535221 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.535780 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:01.535793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:01.537700 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
W20260924 20:14:01.537992 6753 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:01.532476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37083 (local address 127.29.83.190:43721)
0924 20:14:01.532615 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:01.532619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:01.532695 (+ 76us) server_negotiation.cc:407] Connection header received
0924 20:14:01.532796 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:01.532798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:01.532841 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:01.532910 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:01.533359 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.533977 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.534781 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:01.535019 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:01.535128 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:01.537860 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:37083: BlockingRecv error: recv got EOF from 127.29.83.131:37083 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:02.159101 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:02.152737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:40789)
0924 20:14:02.153101 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:02.153116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:02.153229 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:02.153511 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:02.153514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:02.153734 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:02.153925 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.153933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.155021 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.155025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.155613 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.155622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.156125 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.156128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.156201 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.156842 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:02.156858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:02.158980 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":81,"threads_started":1}
W20260924 20:14:02.159319 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:02.152847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:40789 (local address 127.29.83.190:43721)
0924 20:14:02.153133 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:02.153137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:02.153173 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:14:02.153301 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:02.153303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:02.153345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:02.153484 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:02.154091 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.154885 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.155747 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.155992 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.156088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.159151 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:40789: BlockingRecv error: recv got EOF from 127.29.83.129:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:02.352447 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:02.346620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:34367)
0924 20:14:02.346959 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:02.346971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:02.347125 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:02.347406 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:02.347409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:02.347621 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:02.347707 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.347712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.348473 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.348479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.349059 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.349068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.349541 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.349544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.349627 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.350181 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:02.350193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:02.352245 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
W20260924 20:14:02.352520 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:02.346708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:34367 (local address 127.29.83.190:43721)
0924 20:14:02.346879 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:02.346884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:02.347070 (+ 186us) server_negotiation.cc:407] Connection header received
0924 20:14:02.347236 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:02.347240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:02.347296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:02.347384 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:02.347828 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.348361 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.349195 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.349422 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.349539 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.352373 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:34367: BlockingRecv error: recv got EOF from 127.29.83.130:34367 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:02 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:02.544190 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:02.538692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:39607)
0924 20:14:02.538989 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:02.539002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:02.539110 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:02.539375 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:02.539378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:02.539592 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:02.539687 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.539693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.540416 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.540419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.540993 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:02.541001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.541445 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.541449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:02.541531 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.542137 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:02.542159 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:02.544052 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
W20260924 20:14:02.544327 6757 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:02.538848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:39607 (local address 127.29.83.190:43721)
0924 20:14:02.538991 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:02.538995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:02.539078 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:14:02.539190 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:02.539192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:02.539238 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:02.539306 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:02.539792 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.540316 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.541093 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:02.541301 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:02.541374 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:02.544198 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:39607: BlockingRecv error: recv got EOF from 127.29.83.131:39607 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:03.165556 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:03.159880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:48525)
0924 20:14:03.160203 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:03.160218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:03.160311 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:03.160623 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:03.160625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:03.160810 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:03.160909 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.160913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.161699 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.161702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.162353 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.162363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.162832 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.162835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.162903 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.163424 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:03.163436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:03.165361 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
W20260924 20:14:03.165614 6761 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:03.160030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:48525 (local address 127.29.83.190:43721)
0924 20:14:03.160271 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:03.160275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:03.160288 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:14:03.160440 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:03.160443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:03.160498 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:03.160581 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:03.161023 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.161578 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.162485 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.162705 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.162786 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.165493 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:48525: BlockingRecv error: recv got EOF from 127.29.83.129:48525 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":58,"threads_started":1}
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:03.358989 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:03.353232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:42289)
0924 20:14:03.353567 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:03.353593 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:03.353731 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:03.354025 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:03.354028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:03.354246 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:03.354335 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.354341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.355124 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.355130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.355696 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.355704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.356168 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.356172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.356256 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.356793 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:03.356806 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:03.358811 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
W20260924 20:14:03.359077 6761 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:03.353365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:42289 (local address 127.29.83.190:43721)
0924 20:14:03.353535 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:03.353540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:03.353669 (+ 129us) server_negotiation.cc:407] Connection header received
0924 20:14:03.353850 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:03.353853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:03.353909 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:03.354002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:03.354448 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.354992 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.355824 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.356057 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.356161 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.358939 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:42289: BlockingRecv error: recv got EOF from 127.29.83.130:42289 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:03 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:03.550786 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:03.545064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:46431)
0924 20:14:03.545344 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:03.545357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:03.545466 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:03.545715 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:03.545718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:03.545912 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:03.546006 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.546012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.546787 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.546790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.547346 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:03.547353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.547868 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.547872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:03.547944 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.548449 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:03.548461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:03.550637 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20260924 20:14:03.550941 6761 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:03.545135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:46431 (local address 127.29.83.190:43721)
0924 20:14:03.545381 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:03.545385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:03.545399 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:14:03.545557 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:03.545559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:03.545604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:03.545676 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:03.546121 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.546668 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.547484 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:03.547730 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:03.547830 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:03.550821 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:46431: BlockingRecv error: recv got EOF from 127.29.83.131:46431 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:04.172259 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:04.166651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:39945)
0924 20:14:04.166977 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:04.166990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:04.167096 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:04.167339 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:04.167341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:04.167518 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:04.167621 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.167627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.168374 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.168376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.168960 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.168969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.169574 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.169577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.169648 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.170303 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:04.170315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:04.172132 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
W20260924 20:14:04.172632 6765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:04.166740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:39945 (local address 127.29.83.190:43721)
0924 20:14:04.167029 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:04.167033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:04.167047 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:14:04.167176 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:04.167179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:04.167223 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:04.167297 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:04.167714 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.168260 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.169110 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.169421 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.169582 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.172398 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:39945: BlockingRecv error: recv got EOF from 127.29.83.129:39945 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:04.366161 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:04.359938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:39441)
0924 20:14:04.360238 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:04.360253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:04.360346 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:04.360715 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:04.360718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:04.360913 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:04.361025 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.361034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.361988 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.361991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.362570 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.362579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.363066 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.363068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.363132 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.363718 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:04.363730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:04.365950 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
W20260924 20:14:04.366394 6765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:04.360163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:39441 (local address 127.29.83.190:43721)
0924 20:14:04.360305 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:04.360309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:04.360323 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:14:04.360485 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:04.360488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:04.360546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:04.360649 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:04.361280 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.361870 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.362727 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.362950 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.363041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.366115 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:39441: BlockingRecv error: recv got EOF from 127.29.83.130:39441 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:04 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:04.557286 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:04.551693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:39361)
0924 20:14:04.551978 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:04.551991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:04.552102 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:04.552399 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:04.552402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:04.552591 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:04.552682 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.552687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.553519 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.553521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.554186 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:04.554195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.554671 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.554674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:04.554734 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.555302 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:04.555313 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:04.557160 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260924 20:14:04.557417 6765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:04.551756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:39361 (local address 127.29.83.190:43721)
0924 20:14:04.551909 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:04.551913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:04.552074 (+ 161us) server_negotiation.cc:407] Connection header received
0924 20:14:04.552186 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:04.552189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:04.552235 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:04.552308 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:04.552830 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.553411 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.554314 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:04.554563 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:04.554661 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:04.557296 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:39361: BlockingRecv error: recv got EOF from 127.29.83.131:39361 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:05.184406 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:05.173267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:55979)
0924 20:14:05.173555 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:05.173568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:05.173673 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:05.173952 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:05.173954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:05.174140 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:05.174250 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.174260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.175128 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.175131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.175765 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.175775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.176281 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.176284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.176355 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.182225 (+ 5870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:05.182239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:05.184266 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
W20260924 20:14:05.184643 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:05.173355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:55979 (local address 127.29.83.190:43721)
0924 20:14:05.173699 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:05.173702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:05.173716 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:14:05.173778 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:05.173781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:05.173824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:05.173911 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:05.174391 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.174975 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.175882 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.176147 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.176235 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.184442 (+ 8207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:55979: BlockingRecv error: recv got EOF from 127.29.83.129:55979 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:05.372491 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:05.366980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:54999)
0924 20:14:05.367244 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:05.367256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:05.367353 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:05.367594 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:05.367595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:05.367771 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:05.367869 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.367874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.368581 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.368584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.369166 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.369173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.369710 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.369712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.369831 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.370378 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:05.370390 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:05.372357 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
W20260924 20:14:05.372655 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:05.367098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:54999 (local address 127.29.83.190:43721)
0924 20:14:05.367235 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:05.367239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:05.367304 (+ 65us) server_negotiation.cc:407] Connection header received
0924 20:14:05.367418 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:05.367421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:05.367463 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:05.367562 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:05.367964 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.368475 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.369284 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.369609 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.369691 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.372488 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:54999: BlockingRecv error: recv got EOF from 127.29.83.130:54999 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:05 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.131@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:05.563984 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:05.558265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.131:52781)
0924 20:14:05.558550 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:05.558572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:05.558700 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:05.558989 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:05.558992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:05.559211 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:05.559300 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.559306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.560093 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.560097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.560671 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:05.560679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.561142 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.561146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:05.561231 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.561747 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:05.561787 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:05.563800 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260924 20:14:05.564177 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:05.558342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:52781 (local address 127.29.83.190:43721)
0924 20:14:05.558497 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:05.558501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:05.558652 (+ 151us) server_negotiation.cc:407] Connection header received
0924 20:14:05.558815 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:05.558818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:05.558872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:05.558969 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:05.559425 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.559962 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.560796 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:05.561026 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:05.561139 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:05.563954 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.131:52781: BlockingRecv error: recv got EOF from 127.29.83.131:52781 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 20:14:06 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:06 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.129@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:06.191380 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:06.185402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.129:55555)
0924 20:14:06.185831 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:06.185853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:06.185951 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:06.186273 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:06.186275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:06.186454 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:06.186549 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.186554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.187317 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.187320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.187894 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.187903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.188322 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.188325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.188387 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.188896 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:06.188908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:06.191162 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
W20260924 20:14:06.191438 6773 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:06.185552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:55555 (local address 127.29.83.190:43721)
0924 20:14:06.185850 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:06.185855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:06.185904 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:14:06.186086 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:06.186089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:06.186139 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:06.186230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:06.186663 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.187201 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.188001 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.188224 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.188302 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.191284 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.129:55555: BlockingRecv error: recv got EOF from 127.29.83.129:55555 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":62,"threads_started":1}
Sep 24 20:14:06 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:14:06 dist-test-slave-52fr krb5kdc[5715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.83.130@KRBTEST.COM for oryx/127.29.83.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:14:06.378628 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:06.373408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:43721 (local address 127.29.83.130:54733)
0924 20:14:06.373680 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:06.373693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:06.373851 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:06.374079 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:06.374081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:06.374253 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:06.374345 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.374350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.375080 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.375083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.375664 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.375672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.376068 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.376070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.376128 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.376670 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:06.376682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:06.378506 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.83.190:43721: Server oryx/127.29.83.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
W20260924 20:14:06.378809 6773 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:06.373489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:54733 (local address 127.29.83.190:43721)
0924 20:14:06.373679 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:06.373683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:06.373794 (+ 111us) server_negotiation.cc:407] Connection header received
0924 20:14:06.373935 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:06.373937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:06.373980 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:06.374043 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:06.374444 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.374978 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.375765 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.375978 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.376061 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.378648 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.83.130:54733: BlockingRecv error: recv got EOF from 127.29.83.130:54733 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 20:14:06.450477 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6298
I20260924 20:14:06.456468 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6450
I20260924 20:14:06.462013 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6582
I20260924 20:14:06.467123 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6228
2026-09-24T20:14:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18050 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T20:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:14:06Z Disabled control of system clock
I20260924 20:14:06.498358 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:39985
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:14:06.608392 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:06.608726 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:06.608762 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:06.612923 6778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:14:06.613000 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:06.613016 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:06.613039 6778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:14:06.613055 6778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:14:06.617934 6778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:39985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:06.619047 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:06.620411 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:06.628602 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:06.628651 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:06.628664 6778 server_base.cc:1034] running on GCE node
W20260924 20:14:06.628602 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:06.629510 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:06.630717 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:06.631942 6778 hybrid_clock.cc:648] HybridClock initialized: now 1790280846631922 us; error 52 us; skew 500 ppm
I20260924 20:14:06.633965 6778 webserver.cc:460] Webserver started at http://127.29.83.190:42199/ using document root <none> and password file <none>
I20260924 20:14:06.634558 6778 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:06.634626 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:06.634852 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:06.636917 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "397fa57c472d4167b6602c232238ab3b"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "0b9e271d85863526e70190206ef04442"
server_key_iv: "0c372d1856582f93c9793e50245e4750"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.637495 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "397fa57c472d4167b6602c232238ab3b"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "0b9e271d85863526e70190206ef04442"
server_key_iv: "0c372d1856582f93c9793e50245e4750"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.641098 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:14:06.643579 6792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:06.644831 6778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:14:06.644971 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "397fa57c472d4167b6602c232238ab3b"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "0b9e271d85863526e70190206ef04442"
server_key_iv: "0c372d1856582f93c9793e50245e4750"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.645088 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:06.665254 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:06.666023 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:06.666218 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:06.673430 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:39985 every 8 connection(s)
I20260924 20:14:06.673431 6778 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:39985
I20260924 20:14:06.674691 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:14:06.678256 6845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:06.683894 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b: Bootstrap starting.
I20260924 20:14:06.684367 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6778
I20260924 20:14:06.684494 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:14:06.684814 30030 external_mini_cluster.cc:1428] Setting key 21b40d37afac1f0ccd2bba0a44da6e68
I20260924 20:14:06.687430 6845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:06.688201 6845 log.cc:824] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:14:06.690443 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b: No bootstrap required, opened a new log
I20260924 20:14:06.694785 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:06.686318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.29.83.190:39985)
0924 20:14:06.686974 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:06.686989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:06.687024 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:14:06.687766 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:06.687786 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:06.688215 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:06.688550 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:06.689076 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.690134 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.691319 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.691830 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.691981 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.692902 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:06.692907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:06.692921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:06.693312 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:06.693371 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:06.693391 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:06.693483 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:06.693662 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 20:14:06.694016 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260924 20:14:06.694712 6845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } }
I20260924 20:14:06.694998 6845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:06.695082 6845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397fa57c472d4167b6602c232238ab3b, State: Initialized, Role: FOLLOWER
I20260924 20:14:06.695514 6845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } }
I20260924 20:14:06.695668 6845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:14:06.695753 6845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:14:06.695894 6845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:06.697053 6845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } }
I20260924 20:14:06.697407 6845 leader_election.cc:302] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 397fa57c472d4167b6602c232238ab3b; no voters:
I20260924 20:14:06.697722 6845 leader_election.cc:288] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:14:06.697914 6850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:14:06.698194 6850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [term 1 LEADER]: Becoming Leader. State: Replica: 397fa57c472d4167b6602c232238ab3b, State: Running, Role: LEADER
I20260924 20:14:06.698523 6850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } }
I20260924 20:14:06.698980 6845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:14:06.700241 6852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397fa57c472d4167b6602c232238ab3b. Latest consensus state: current_term: 1 leader_uuid: "397fa57c472d4167b6602c232238ab3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } } }
I20260924 20:14:06.700415 6852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:06.700241 6851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "397fa57c472d4167b6602c232238ab3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "397fa57c472d4167b6602c232238ab3b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 39985 } } }
I20260924 20:14:06.700511 6851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:06.700904 6860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:14:06.703817 6860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:14:06.709209 6860 catalog_manager.cc:1347] Generated new cluster ID: 0a50381d8a8f4eb190bb9ccbceda24c7
I20260924 20:14:06.709290 6860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:14:06.717137 6860 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:14:06.718469 6860 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:14:06.733428 6860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 397fa57c472d4167b6602c232238ab3b: Generated new TSK 0
I20260924 20:14:06.734303 6860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:14:06.742602 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--builtin_ntp_servers=127.29.83.148:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 20:14:06.861385 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:06.861652 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:06.861690 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:06.865653 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:06.865710 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:06.865823 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:14:06.870513 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:06.871570 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:06.872895 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:06.882799 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:06.882807 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:06.882809 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:06.883147 6869 server_base.cc:1034] running on GCE node
I20260924 20:14:06.883597 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:06.884308 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:06.885510 6869 hybrid_clock.cc:648] HybridClock initialized: now 1790280846885472 us; error 50 us; skew 500 ppm
I20260924 20:14:06.887612 6869 webserver.cc:460] Webserver started at http://127.29.83.129:34449/ using document root <none> and password file <none>
I20260924 20:14:06.888167 6869 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:06.888227 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:06.888393 6869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:06.890583 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "cec45219cee9456c811a006a9646a78e"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "101e47f8955f379bf623b71fb9dfd67d"
server_key_iv: "7f83fc64afebbecb52670cfd7986faea"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.891121 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "cec45219cee9456c811a006a9646a78e"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "101e47f8955f379bf623b71fb9dfd67d"
server_key_iv: "7f83fc64afebbecb52670cfd7986faea"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.894867 6869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:14:06.897439 6883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:06.898767 6869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:14:06.898864 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "cec45219cee9456c811a006a9646a78e"
format_stamp: "Formatted at 2026-09-24 20:14:06 on dist-test-slave-52fr"
server_key: "101e47f8955f379bf623b71fb9dfd67d"
server_key_iv: "7f83fc64afebbecb52670cfd7986faea"
server_key_version: "encryptionkey@0"
I20260924 20:14:06.898957 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:06.935766 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:06.936456 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:06.936615 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:06.937172 6869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:06.938162 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:06.938210 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:06.938249 6869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:06.938264 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:06.947052 6869 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45149
I20260924 20:14:06.947093 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45149 every 8 connection(s)
I20260924 20:14:06.947983 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:14:06.950397 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 6869
I20260924 20:14:06.950481 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:14:06.950790 30030 external_mini_cluster.cc:1428] Setting key 3a346dd2bf751db1dc099d3593f5fc57
I20260924 20:14:06.952912 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--builtin_ntp_servers=127.29.83.148:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:14:06.958364 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:06.950005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:46359 (local address 127.29.83.190:39985)
0924 20:14:06.950146 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:06.950150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:06.951342 (+ 1192us) server_negotiation.cc:407] Connection header received
0924 20:14:06.952889 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:06.952893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:06.952961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:06.953063 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:06.954145 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.954690 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.955886 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.956287 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.956409 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.957664 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:06.957671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:06.957677 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:06.957698 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:06.957725 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:06.957740 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:06.957850 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:06.958056 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 20:14:06.958137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:14:06.960216 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:06.950359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:39985 (local address 127.29.83.129:46359)
0924 20:14:06.951167 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:06.951221 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:06.952664 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:06.953333 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:06.953341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:06.953438 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:06.953986 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.954001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.954822 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.954825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.955755 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:06.955764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:06.956472 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:06.956476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:06.956635 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:06.957384 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:06.957386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:06.957429 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:06.957499 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:06.957896 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:06.957905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:06.957910 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:14:06.958033 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 20:14:06.958276 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
I20260924 20:14:06.961580 6997 heartbeater.cc:347] Connected to a master server at 127.29.83.190:39985
I20260924 20:14:06.961905 6997 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:06.962634 6997 heartbeater.cc:511] Master 127.29.83.190:39985 requested a full tablet report, sending...
I20260924 20:14:06.964299 6809 ts_manager.cc:194] Registered new tserver with Master: cec45219cee9456c811a006a9646a78e (127.29.83.129:45149)
I20260924 20:14:06.965479 6809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.129:46359
W20260924 20:14:07.067051 7000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:07.067327 7000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:07.067365 7000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:07.071557 7000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:07.071633 7000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:07.071689 7000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:14:07.076560 7000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:07.077689 7000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:07.079108 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:07.088896 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.089087 7000 server_base.cc:1034] running on GCE node
W20260924 20:14:07.089183 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.089183 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.089730 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:07.090413 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:07.091590 7000 hybrid_clock.cc:648] HybridClock initialized: now 1790280847091550 us; error 48 us; skew 500 ppm
I20260924 20:14:07.093464 7000 webserver.cc:460] Webserver started at http://127.29.83.130:45259/ using document root <none> and password file <none>
I20260924 20:14:07.094084 7000 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:07.094153 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:07.094319 7000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:07.096304 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "9a011cfce3a3432cb3e49586ddb00c1d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "bf1b80b6a055d63a48ee2a9edadc0280"
server_key_iv: "ef1deba07d3fff7107ed3992689dc67c"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.096809 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "9a011cfce3a3432cb3e49586ddb00c1d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "bf1b80b6a055d63a48ee2a9edadc0280"
server_key_iv: "ef1deba07d3fff7107ed3992689dc67c"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.100780 7000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:14:07.103591 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.104914 7000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:14:07.105067 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "9a011cfce3a3432cb3e49586ddb00c1d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "bf1b80b6a055d63a48ee2a9edadc0280"
server_key_iv: "ef1deba07d3fff7107ed3992689dc67c"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.105192 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:07.122061 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:07.122804 7000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:07.123021 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:07.123648 7000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:07.124603 7000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:07.124648 7000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.124719 7000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:07.124758 7000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.133970 7000 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:38905
I20260924 20:14:07.134061 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:38905 every 8 connection(s)
I20260924 20:14:07.134950 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:14:07.140558 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7000
I20260924 20:14:07.140681 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:14:07.140978 30030 external_mini_cluster.cc:1428] Setting key 9531aa9c8a7ffc1062c400b4f0f628aa
I20260924 20:14:07.143316 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--builtin_ntp_servers=127.29.83.148:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:14:07.145362 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.136891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:59893 (local address 127.29.83.190:39985)
0924 20:14:07.137036 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.137040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.138157 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 20:14:07.139498 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.139501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.139552 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.139629 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.140605 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.141378 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.143008 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.143375 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.143513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:07.144892 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:07.144895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:07.144897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:07.144910 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:07.144929 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:07.144937 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:07.145004 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:07.145197 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 20:14:07.145251 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:14:07.146490 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.137225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:39985 (local address 127.29.83.130:59893)
0924 20:14:07.137990 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:07.138034 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:07.139301 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:07.139806 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:07.139813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:07.139899 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:07.140422 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:07.140437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.141584 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.141589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:07.142754 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:07.142764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.143655 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.143659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:07.143829 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:07.144540 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:07.144544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:07.144616 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:07.144717 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:07.145033 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:07.145044 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:07.145051 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:14:07.145193 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 20:14:07.145546 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260924 20:14:07.147877 7128 heartbeater.cc:347] Connected to a master server at 127.29.83.190:39985
I20260924 20:14:07.148177 7128 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:07.148773 7128 heartbeater.cc:511] Master 127.29.83.190:39985 requested a full tablet report, sending...
I20260924 20:14:07.150012 6809 ts_manager.cc:194] Registered new tserver with Master: 9a011cfce3a3432cb3e49586ddb00c1d (127.29.83.130:38905)
I20260924 20:14:07.150902 6809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.130:59893
W20260924 20:14:07.258150 7131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:07.258424 7131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:07.258463 7131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:07.262636 7131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:07.262702 7131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:07.262758 7131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:14:07.268142 7131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:39985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:07.269313 7131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:07.270773 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:07.280804 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.280773 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.280814 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.281172 7131 server_base.cc:1034] running on GCE node
I20260924 20:14:07.281589 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:07.282326 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:07.283524 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790280847283482 us; error 53 us; skew 500 ppm
I20260924 20:14:07.285676 7131 webserver.cc:460] Webserver started at http://127.29.83.131:43375/ using document root <none> and password file <none>
I20260924 20:14:07.286365 7131 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:07.286463 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:07.286706 7131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:07.288878 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "1d4cbf13c4ed422895f777ce69cbb80f"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "fe24d73a995c318ff3454682a42b89c8"
server_key_iv: "c7182c92fe0f829ef2c0448d35bacbbc"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.289521 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "1d4cbf13c4ed422895f777ce69cbb80f"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "fe24d73a995c318ff3454682a42b89c8"
server_key_iv: "c7182c92fe0f829ef2c0448d35bacbbc"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.293396 7131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:14:07.296356 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.297906 7131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:14:07.298064 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "1d4cbf13c4ed422895f777ce69cbb80f"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "fe24d73a995c318ff3454682a42b89c8"
server_key_iv: "c7182c92fe0f829ef2c0448d35bacbbc"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.298197 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:07.322049 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:07.322861 7131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:07.323076 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:07.323694 7131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:07.324671 7131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:07.324745 7131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.324818 7131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:07.324860 7131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.334344 7131 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:43033
I20260924 20:14:07.334489 7258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:43033 every 8 connection(s)
I20260924 20:14:07.335661 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:14:07.340617 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7131
I20260924 20:14:07.340744 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:14:07.341099 30030 external_mini_cluster.cc:1428] Setting key d40efd10b3761ba5d96f6ca88e01a3e2
I20260924 20:14:07.347074 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.339438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:41975 (local address 127.29.83.190:39985)
0924 20:14:07.339578 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.339583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.339643 (+ 60us) server_negotiation.cc:407] Connection header received
0924 20:14:07.341661 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.341665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.341723 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.341837 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.343148 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.343873 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.345034 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.345258 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.345343 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:07.346523 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:07.346526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:07.346528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:07.346539 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:07.346558 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:07.346566 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:07.346634 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:07.346903 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 20:14:07.346959 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:14:07.347954 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.338337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:39985 (local address 127.29.83.131:41975)
0924 20:14:07.339491 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:07.339551 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:07.341178 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:07.342172 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:07.342185 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:07.342307 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:07.342968 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:07.342986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.344025 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.344028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:07.344914 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:07.344922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:07.345389 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:07.345391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:07.345565 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:07.346234 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:07.346236 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:07.346277 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:07.346339 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:07.346719 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:07.346727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:07.346732 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:14:07.346860 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 20:14:07.347084 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":134,"threads_started":1}
I20260924 20:14:07.349233 7259 heartbeater.cc:347] Connected to a master server at 127.29.83.190:39985
I20260924 20:14:07.349499 7259 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:07.350117 7259 heartbeater.cc:511] Master 127.29.83.190:39985 requested a full tablet report, sending...
I20260924 20:14:07.351157 6809 ts_manager.cc:194] Registered new tserver with Master: 1d4cbf13c4ed422895f777ce69cbb80f (127.29.83.131:43033)
I20260924 20:14:07.351729 6809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.131:41975
I20260924 20:14:07.355599 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:14:07.358795 6848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:07.357624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.29.83.190:39985)
0924 20:14:07.357778 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.357784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.358102 (+ 318us) server_negotiation.cc:407] Connection header received
0924 20:14:07.358174 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.358176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.358216 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.358297 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.358625 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42456: BlockingRecv error: recv got EOF from 127.0.0.1:42456 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:14:07.358969 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6869
I20260924 20:14:07.364856 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7000
I20260924 20:14:07.370157 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7131
I20260924 20:14:07.375607 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 6778
2026-09-24T20:14:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (906 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T20:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:14:07Z Disabled control of system clock
I20260924 20:14:07.404763 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40813
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:14:07.515223 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:07.515496 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:07.515539 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:07.519961 7270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:14:07.520036 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:07.520052 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:07.520071 7270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:14:07.520087 7270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:14:07.525152 7270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:07.526335 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:07.527734 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:07.536469 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.536489 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.536536 7270 server_base.cc:1034] running on GCE node
W20260924 20:14:07.536476 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.537382 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:07.538604 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:07.539853 7270 hybrid_clock.cc:648] HybridClock initialized: now 1790280847539825 us; error 56 us; skew 500 ppm
I20260924 20:14:07.541898 7270 webserver.cc:460] Webserver started at http://127.29.83.190:42627/ using document root <none> and password file <none>
I20260924 20:14:07.542492 7270 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:07.542588 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:07.542838 7270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:07.544934 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "6114856938144f949806d6071d932a2d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "0f96f29a00d4e3e03a60c405892a90d3"
server_key_iv: "429236334cb20a4356b9e6bd1da4d3b4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.545507 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "6114856938144f949806d6071d932a2d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "0f96f29a00d4e3e03a60c405892a90d3"
server_key_iv: "429236334cb20a4356b9e6bd1da4d3b4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.549229 7270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:14:07.551833 7284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.553176 7270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:14:07.553342 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "6114856938144f949806d6071d932a2d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "0f96f29a00d4e3e03a60c405892a90d3"
server_key_iv: "429236334cb20a4356b9e6bd1da4d3b4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.553478 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:07.558249 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:07.558926 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:07.559127 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:07.565676 7270 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40813
I20260924 20:14:07.565860 7336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40813 every 8 connection(s)
I20260924 20:14:07.566905 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:14:07.570559 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7270
I20260924 20:14:07.570698 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:14:07.570791 7337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:07.571014 30030 external_mini_cluster.cc:1428] Setting key 25bcd8b02afec9ca104aee2fa300baf9
I20260924 20:14:07.576696 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.572494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.29.83.190:40813)
0924 20:14:07.573205 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.573220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.573255 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:14:07.573978 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.573996 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.574362 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.574722 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.575459 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:07.575464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:07.575475 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:07.575485 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:07.575538 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:07.575560 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:07.575712 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:07.575780 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 20:14:07.576021 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":150,"threads_started":1}
I20260924 20:14:07.577889 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d: Bootstrap starting.
I20260924 20:14:07.580123 7337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:07.580852 7337 log.cc:824] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:14:07.582826 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d: No bootstrap required, opened a new log
I20260924 20:14:07.586669 7337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } }
I20260924 20:14:07.586952 7337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:07.587033 7337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6114856938144f949806d6071d932a2d, State: Initialized, Role: FOLLOWER
I20260924 20:14:07.587514 7337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } }
I20260924 20:14:07.587666 7337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:14:07.587752 7337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:14:07.587855 7337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:07.588778 7337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } }
I20260924 20:14:07.589169 7337 leader_election.cc:302] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6114856938144f949806d6071d932a2d; no voters:
I20260924 20:14:07.589535 7337 leader_election.cc:288] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:14:07.589671 7342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:14:07.589923 7342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [term 1 LEADER]: Becoming Leader. State: Replica: 6114856938144f949806d6071d932a2d, State: Running, Role: LEADER
I20260924 20:14:07.590252 7342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } }
I20260924 20:14:07.590941 7337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:14:07.592212 7343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6114856938144f949806d6071d932a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } } }
I20260924 20:14:07.592212 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6114856938144f949806d6071d932a2d. Latest consensus state: current_term: 1 leader_uuid: "6114856938144f949806d6071d932a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6114856938144f949806d6071d932a2d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40813 } } }
I20260924 20:14:07.592427 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:07.592363 7343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:07.593148 7351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:14:07.596314 7351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:14:07.601617 7351 catalog_manager.cc:1347] Generated new cluster ID: d1b8c92e016d4f518ede8d5aab8ec401
I20260924 20:14:07.601691 7351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:14:07.611047 7351 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:14:07.611388 7351 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:14:07.628150 7351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6114856938144f949806d6071d932a2d: Generated new TSK 0
I20260924 20:14:07.628974 7351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:14:07.637976 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--builtin_ntp_servers=127.29.83.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:14:07.757740 7361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:07.758083 7361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:07.758126 7361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:07.762216 7361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:07.762280 7361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:07.762338 7361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:14:07.767158 7361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:07.768188 7361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:07.769543 7361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:07.779778 7369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.779757 7366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.779804 7361 server_base.cc:1034] running on GCE node
W20260924 20:14:07.779757 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.780392 7361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:07.781013 7361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:07.782174 7361 hybrid_clock.cc:648] HybridClock initialized: now 1790280847782160 us; error 26 us; skew 500 ppm
I20260924 20:14:07.783931 7361 webserver.cc:460] Webserver started at http://127.29.83.129:44355/ using document root <none> and password file <none>
I20260924 20:14:07.784521 7361 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:07.784590 7361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:07.784798 7361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:07.786872 7361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "96c5e113e39f44eeab9fc024165bd64d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "508210164ec7a7f29ad63d282d7cb17a"
server_key_iv: "49121eadcc309956a216a8c892d8a23a"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.787433 7361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "96c5e113e39f44eeab9fc024165bd64d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "508210164ec7a7f29ad63d282d7cb17a"
server_key_iv: "49121eadcc309956a216a8c892d8a23a"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.791090 7361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:14:07.793871 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.795141 7361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:14:07.795265 7361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "96c5e113e39f44eeab9fc024165bd64d"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "508210164ec7a7f29ad63d282d7cb17a"
server_key_iv: "49121eadcc309956a216a8c892d8a23a"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.795382 7361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:07.799824 7361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:07.800426 7361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:07.800611 7361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:07.801142 7361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:07.802119 7361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:07.802165 7361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.802202 7361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:07.802240 7361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.811319 7361 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:46463
I20260924 20:14:07.811407 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:46463 every 8 connection(s)
I20260924 20:14:07.812273 7361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:14:07.814819 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7361
I20260924 20:14:07.814913 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:14:07.815204 30030 external_mini_cluster.cc:1428] Setting key 7aa83a3c64ed8dd8b0fc170207569b50
I20260924 20:14:07.817123 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--builtin_ntp_servers=127.29.83.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 20:14:07.819236 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.814468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:55403 (local address 127.29.83.190:40813)
0924 20:14:07.814694 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.814701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.815940 (+ 1239us) server_negotiation.cc:407] Connection header received
0924 20:14:07.817306 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.817309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.817349 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.817440 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.818688 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:07.818694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:07.818698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:07.818716 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:07.818745 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:07.818760 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:07.818867 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:07.819017 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 20:14:07.819069 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 20:14:07.820082 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.815016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40813 (local address 127.29.83.129:55403)
0924 20:14:07.815776 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:07.815829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:07.817091 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:07.817620 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:07.817627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:07.817720 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:07.818399 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:07.818401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:07.818451 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:07.818524 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:07.818865 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:07.818874 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:07.818879 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:14:07.819042 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 20:14:07.819239 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260924 20:14:07.821496 7489 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40813
I20260924 20:14:07.821727 7489 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:07.822178 7489 heartbeater.cc:511] Master 127.29.83.190:40813 requested a full tablet report, sending...
I20260924 20:14:07.823720 7301 ts_manager.cc:194] Registered new tserver with Master: 96c5e113e39f44eeab9fc024165bd64d (127.29.83.129:46463)
W20260924 20:14:07.933036 7492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:07.933400 7492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:07.933499 7492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:07.937750 7492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:07.937882 7492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:07.937984 7492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:14:07.943032 7492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:07.944221 7492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:07.945710 7492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:07.956204 7498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.956218 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:07.956275 7497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:07.956415 7492 server_base.cc:1034] running on GCE node
I20260924 20:14:07.956957 7492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:07.957681 7492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:07.958880 7492 hybrid_clock.cc:648] HybridClock initialized: now 1790280847958842 us; error 50 us; skew 500 ppm
I20260924 20:14:07.960875 7492 webserver.cc:460] Webserver started at http://127.29.83.130:44873/ using document root <none> and password file <none>
I20260924 20:14:07.961472 7492 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:07.961573 7492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:07.961854 7492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:07.963994 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "e00acf8e7c0348778fbe8e4153f745bd"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "3dfdfc3b859ecf8062731bee106cc038"
server_key_iv: "d437fad8a862aafb1d88feb9ee7c52e4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.964567 7492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "e00acf8e7c0348778fbe8e4153f745bd"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "3dfdfc3b859ecf8062731bee106cc038"
server_key_iv: "d437fad8a862aafb1d88feb9ee7c52e4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.968231 7492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:14:07.970821 7506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.972074 7492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:14:07.972208 7492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "e00acf8e7c0348778fbe8e4153f745bd"
format_stamp: "Formatted at 2026-09-24 20:14:07 on dist-test-slave-52fr"
server_key: "3dfdfc3b859ecf8062731bee106cc038"
server_key_iv: "d437fad8a862aafb1d88feb9ee7c52e4"
server_key_version: "encryptionkey@0"
I20260924 20:14:07.972326 7492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:07.976941 7492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:07.977563 7492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:07.977746 7492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:07.978312 7492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:07.979238 7492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:07.979308 7492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.979378 7492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:07.979422 7492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:07.987970 7492 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:41357
I20260924 20:14:07.988005 7619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:41357 every 8 connection(s)
I20260924 20:14:07.988857 7492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:14:07.994382 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7492
I20260924 20:14:07.994525 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:14:07.994855 30030 external_mini_cluster.cc:1428] Setting key 17d7d611afb4e5aa485931c43a46ea12
I20260924 20:14:07.995641 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.990826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:36017 (local address 127.29.83.190:40813)
0924 20:14:07.990988 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:07.990993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:07.992164 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 20:14:07.993608 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:07.993611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:07.993649 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:07.993730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:07.995136 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:07.995138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:07.995140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:07.995152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:07.995174 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:07.995182 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:07.995259 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:07.995475 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 20:14:07.995534 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:14:07.996630 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:07.991257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40813 (local address 127.29.83.130:36017)
0924 20:14:07.991979 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:07.992021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:07.993384 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:07.993966 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:07.993973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:07.994065 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:07.994811 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:07.994814 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:07.994869 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:07.994964 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:07.995270 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:07.995281 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:07.995287 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:14:07.995439 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 20:14:07.995712 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260924 20:14:07.997087 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--builtin_ntp_servers=127.29.83.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 20:14:07.997931 7620 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40813
I20260924 20:14:07.998163 7620 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:07.998611 7620 heartbeater.cc:511] Master 127.29.83.190:40813 requested a full tablet report, sending...
I20260924 20:14:07.999738 7301 ts_manager.cc:194] Registered new tserver with Master: e00acf8e7c0348778fbe8e4153f745bd (127.29.83.130:41357)
W20260924 20:14:08.110908 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:08.111192 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:08.111233 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:08.115363 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:08.115432 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:08.115491 7623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:14:08.120250 7623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:08.121236 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:08.122608 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:08.131847 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:08.131837 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:08.131896 7623 server_base.cc:1034] running on GCE node
W20260924 20:14:08.131837 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:08.132586 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:08.133122 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:08.134270 7623 hybrid_clock.cc:648] HybridClock initialized: now 1790280848134236 us; error 46 us; skew 500 ppm
I20260924 20:14:08.136011 7623 webserver.cc:460] Webserver started at http://127.29.83.131:46717/ using document root <none> and password file <none>
I20260924 20:14:08.136571 7623 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:08.136631 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:08.136857 7623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:08.138909 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "3f4b53ec01d1480693b8f0f4bccc5cfc"
format_stamp: "Formatted at 2026-09-24 20:14:08 on dist-test-slave-52fr"
server_key: "2b7a58be8c83578a6228bbeb8b8108a3"
server_key_iv: "2ea39b0e823459c59ad17408cad40683"
server_key_version: "encryptionkey@0"
I20260924 20:14:08.139475 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4b53ec01d1480693b8f0f4bccc5cfc"
format_stamp: "Formatted at 2026-09-24 20:14:08 on dist-test-slave-52fr"
server_key: "2b7a58be8c83578a6228bbeb8b8108a3"
server_key_iv: "2ea39b0e823459c59ad17408cad40683"
server_key_version: "encryptionkey@0"
I20260924 20:14:08.143035 7623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:14:08.145643 7637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:08.146958 7623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:14:08.147100 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "3f4b53ec01d1480693b8f0f4bccc5cfc"
format_stamp: "Formatted at 2026-09-24 20:14:08 on dist-test-slave-52fr"
server_key: "2b7a58be8c83578a6228bbeb8b8108a3"
server_key_iv: "2ea39b0e823459c59ad17408cad40683"
server_key_version: "encryptionkey@0"
I20260924 20:14:08.147225 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:08.151985 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:08.152606 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:08.152788 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:08.153363 7623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:08.154382 7623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:08.154440 7623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:08.154502 7623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:08.154536 7623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:08.163640 7623 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:44115
I20260924 20:14:08.163661 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:44115 every 8 connection(s)
I20260924 20:14:08.164619 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:14:08.170778 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:08.166507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:40173 (local address 127.29.83.190:40813)
0924 20:14:08.166645 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:08.166649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:08.167692 (+ 1043us) server_negotiation.cc:407] Connection header received
0924 20:14:08.168985 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:08.168988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:08.169041 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:08.169166 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:08.170303 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:08.170306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:08.170309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:14:08.170322 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:08.170343 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:14:08.170354 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:08.170465 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:08.170600 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 20:14:08.170645 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:14:08.171609 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:08.166866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40813 (local address 127.29.83.131:40173)
0924 20:14:08.167539 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:08.167580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:08.168769 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:08.169330 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:08.169338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:08.169429 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:08.170045 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:14:08.170047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:08.170087 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:14:08.170154 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:08.170451 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:08.170459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:08.170464 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:14:08.170642 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 20:14:08.170876 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260924 20:14:08.172690 7751 heartbeater.cc:347] Connected to a master server at 127.29.83.190:40813
I20260924 20:14:08.172914 7751 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:08.173298 7751 heartbeater.cc:511] Master 127.29.83.190:40813 requested a full tablet report, sending...
I20260924 20:14:08.174199 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7623
I20260924 20:14:08.174305 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:14:08.174409 7301 ts_manager.cc:194] Registered new tserver with Master: 3f4b53ec01d1480693b8f0f4bccc5cfc (127.29.83.131:44115)
I20260924 20:14:08.174613 30030 external_mini_cluster.cc:1428] Setting key 01507294a6a97da0480291c1a1ab2289
I20260924 20:14:08.176515 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:14:08.179265 7340 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:14:08.178386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.29.83.190:40813)
0924 20:14:08.178494 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:08.178500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:08.178696 (+ 196us) server_negotiation.cc:407] Connection header received
0924 20:14:08.178786 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:08.178788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:08.178817 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:08.178874 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:08.179131 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41858: BlockingRecv error: recv got EOF from 127.0.0.1:41858 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:14:08.179706 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7361
I20260924 20:14:08.185820 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7492
I20260924 20:14:08.191978 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7623
I20260924 20:14:08.197701 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7270
2026-09-24T20:14:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:14:08 dist-test-slave-52fr krb5kdc[7761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:14:08 dist-test-slave-52fr krb5kdc[7761](info): set up 2 sockets
Sep 24 20:14:08 dist-test-slave-52fr krb5kdc[7761](info): commencing operation
krb5kdc: starting...
W20260924 20:14:09.246829 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:14:09 dist-test-slave-52fr krb5kdc[7761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:14:09.369354 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46113
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:14:09.483258 7774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:09.483641 7774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:09.483737 7774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:09.488032 7774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:14:09.488137 7774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:09.488173 7774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:09.488212 7774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:14:09.488246 7774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:14:09.494349 7774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:09.495605 7774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:09.496735 7774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:14:09.505661 7774 server_base.cc:1034] running on GCE node
W20260924 20:14:09.505595 7780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:09.505609 7782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:09.505595 7779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:09.506496 7774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:09.507576 7774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:09.508740 7774 hybrid_clock.cc:648] HybridClock initialized: now 1790280849508731 us; error 27 us; skew 500 ppm
Sep 24 20:14:09 dist-test-slave-52fr krb5kdc[7761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280849, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:14:09.511495 7774 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:14:09.512439 7774 webserver.cc:460] Webserver started at http://127.29.83.190:46705/ using document root <none> and password file <none>
I20260924 20:14:09.512995 7774 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:09.513062 7774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:09.513293 7774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:09.515391 7774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "54a8acf72a327eb52c328d0c3ab443a2"
server_key_iv: "153469f6634ab0c3adff8948be87353e"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.515961 7774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "54a8acf72a327eb52c328d0c3ab443a2"
server_key_iv: "153469f6634ab0c3adff8948be87353e"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.519706 7774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:14:09.522344 7789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:09.523767 7774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:14:09.523927 7774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "54a8acf72a327eb52c328d0c3ab443a2"
server_key_iv: "153469f6634ab0c3adff8948be87353e"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.524055 7774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:09.540566 7774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:09.543555 7774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:09.543766 7774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:09.551687 7774 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:46113
I20260924 20:14:09.551767 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:46113 every 8 connection(s)
I20260924 20:14:09.552735 7774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:14:09.555617 7842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:09.556061 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7774
I20260924 20:14:09.556185 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:14:09.556493 30030 external_mini_cluster.cc:1428] Setting key 7e8286dd0018549f0618a726109e6988
I20260924 20:14:09.562040 7842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: Bootstrap starting.
I20260924 20:14:09.565066 7842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:09.566023 7842 log.cc:824] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:14:09 dist-test-slave-52fr krb5kdc[7761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:14:09.568261 7842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: No bootstrap required, opened a new log
I20260924 20:14:09.571651 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:09.558005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.29.83.190:46113)
0924 20:14:09.558464 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:09.558480 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:09.558518 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:14:09.559192 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:09.559209 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:09.559545 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:09.559839 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:09.560737 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.561811 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.563001 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.563535 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.563672 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:09.566688 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:09.566693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:09.566708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:14:09.566740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:09.568658 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:09.569147 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:09.569153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:09.569158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:09.569273 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:09.569587 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:09.569590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:09.569592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:09.570068 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:09.570294 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:09.570583 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 20:14:09.570884 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":157,"threads_started":1}
I20260924 20:14:09.572099 7842 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } }
I20260924 20:14:09.572355 7842 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:09.572434 7842 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2fd0c9ba3b4604b7bfc2a7c286a86b, State: Initialized, Role: FOLLOWER
I20260924 20:14:09.572922 7842 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } }
I20260924 20:14:09.573089 7842 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:14:09.573177 7842 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:14:09.573279 7842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:09.574456 7842 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } }
I20260924 20:14:09.574805 7842 leader_election.cc:302] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce2fd0c9ba3b4604b7bfc2a7c286a86b; no voters:
I20260924 20:14:09.575129 7842 leader_election.cc:288] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:14:09.575296 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:14:09.575544 7847 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [term 1 LEADER]: Becoming Leader. State: Replica: ce2fd0c9ba3b4604b7bfc2a7c286a86b, State: Running, Role: LEADER
I20260924 20:14:09.575857 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } }
I20260924 20:14:09.576489 7842 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:14:09.577046 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce2fd0c9ba3b4604b7bfc2a7c286a86b. Latest consensus state: current_term: 1 leader_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } } }
I20260924 20:14:09.577257 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:09.577726 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2fd0c9ba3b4604b7bfc2a7c286a86b" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46113 } } }
I20260924 20:14:09.577957 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b [sys.catalog]: This master's current role is: LEADER
I20260924 20:14:09.579236 7862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:14:09.582301 7862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:14:09.588294 7862 catalog_manager.cc:1347] Generated new cluster ID: cefa3185745841419770439c820d3850
I20260924 20:14:09.588380 7862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:14:09.608640 7862 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:14:09.609714 7862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:14:09.615036 7862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: Generated new TSK 0
I20260924 20:14:09.615770 7862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:14:09.654178 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:14:09.768950 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:09.769275 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:09.769322 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:09.773531 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:09.773612 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:09.773674 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:14:09.779488 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:09.780577 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:09.781623 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:09.791411 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:09.791440 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:09.791384 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:09.791481 7868 server_base.cc:1034] running on GCE node
I20260924 20:14:09.792156 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:09.792739 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:09.793943 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790280849793907 us; error 48 us; skew 500 ppm
Sep 24 20:14:09 dist-test-slave-52fr krb5kdc[7761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280849, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:14:09.796921 7868 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:14:09.797883 7868 webserver.cc:460] Webserver started at http://127.29.83.129:38531/ using document root <none> and password file <none>
I20260924 20:14:09.798454 7868 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:09.798549 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:09.798787 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:09.800823 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "6459fe97944742f38c1b86fcbe6d1e94"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "86e83e02775f4afe59249d2d9a99a64e"
server_key_iv: "fc8e66d5d85bc2777856e44e0115cd09"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.801434 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "6459fe97944742f38c1b86fcbe6d1e94"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "86e83e02775f4afe59249d2d9a99a64e"
server_key_iv: "fc8e66d5d85bc2777856e44e0115cd09"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.805413 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:14:09.808000 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:09.809185 7868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:14:09.809317 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "6459fe97944742f38c1b86fcbe6d1e94"
format_stamp: "Formatted at 2026-09-24 20:14:09 on dist-test-slave-52fr"
server_key: "86e83e02775f4afe59249d2d9a99a64e"
server_key_iv: "fc8e66d5d85bc2777856e44e0115cd09"
server_key_version: "encryptionkey@0"
I20260924 20:14:09.809432 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:09.822654 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:09.826130 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:09.826417 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:09.827081 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:09.827986 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:09.828065 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:09.828161 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:09.828212 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:09.837283 7868 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45587
I20260924 20:14:09.837306 7996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45587 every 8 connection(s)
I20260924 20:14:09.838279 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:14:09.840773 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 7868
I20260924 20:14:09.840907 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:14:09.841240 30030 external_mini_cluster.cc:1428] Setting key acc214285d7560d4730eb707b0b38c64
Sep 24 20:14:09 dist-test-slave-52fr krb5kdc[7761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280849, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:14:09.852193 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:09.840167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:38037 (local address 127.29.83.190:46113)
0924 20:14:09.840304 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:09.840313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:09.841150 (+ 837us) server_negotiation.cc:407] Connection header received
0924 20:14:09.842739 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:09.842742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:09.842792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:09.842895 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:09.844036 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.844566 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.845727 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.846023 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.846135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:09.848555 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:09.848558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:09.848560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:14:09.848583 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:09.850169 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:09.850669 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:09.850673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:09.850675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:09.850755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:09.851101 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:09.851105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:09.851107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:09.851289 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:09.851393 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:09.851896 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 20:14:09.852026 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:14:09.853111 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:09.840510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46113 (local address 127.29.83.129:38037)
0924 20:14:09.840983 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:09.841034 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:09.842521 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:09.843078 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:09.843085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:09.843434 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:09.843889 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:09.843901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.844697 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.844701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:09.845597 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:09.845607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:09.846173 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:09.846178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:09.846356 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:09.846966 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:09.846983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:09.848374 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:09.850288 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:09.850294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:09.850309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:09.850566 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:09.850855 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:09.850858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:09.850860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:09.851006 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:09.851403 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:09.851411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:09.851665 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:14:09.851858 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 20:14:09.852203 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260924 20:14:09.854830 7997 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46113
I20260924 20:14:09.855126 7997 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:09.855703 7997 heartbeater.cc:511] Master 127.29.83.190:46113 requested a full tablet report, sending...
I20260924 20:14:09.857681 7806 ts_manager.cc:194] Registered new tserver with Master: 6459fe97944742f38c1b86fcbe6d1e94 (127.29.83.129:45587)
I20260924 20:14:09.858922 7806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:38037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:14:09.869040 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:14:09.983019 8002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:09.983300 8002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:09.983345 8002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:09.987721 8002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:09.987800 8002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:09.987891 8002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:14:09.993194 8002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:09.994297 8002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:09.995374 8002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:10.005345 8007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:10.005407 8002 server_base.cc:1034] running on GCE node
W20260924 20:14:10.005354 8008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:10.005364 8010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:10.006093 8002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:10.006763 8002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:10.007961 8002 hybrid_clock.cc:648] HybridClock initialized: now 1790280850007923 us; error 51 us; skew 500 ppm
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[7761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280850, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:14:10.011211 8002 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:14:10.012228 8002 webserver.cc:460] Webserver started at http://127.29.83.130:43075/ using document root <none> and password file <none>
I20260924 20:14:10.012952 8002 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:10.013046 8002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:10.013294 8002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:10.015450 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "55bec0a615ea44f192222db892f2bbee"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "8baa22baf33d9c803635d3fd25e41509"
server_key_iv: "edcbb284ad1973d2a342344d77c6ee24"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.016021 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "55bec0a615ea44f192222db892f2bbee"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "8baa22baf33d9c803635d3fd25e41509"
server_key_iv: "edcbb284ad1973d2a342344d77c6ee24"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.020733 8002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:14:10.024082 8017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.025791 8002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:14:10.025903 8002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "55bec0a615ea44f192222db892f2bbee"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "8baa22baf33d9c803635d3fd25e41509"
server_key_iv: "edcbb284ad1973d2a342344d77c6ee24"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.026026 8002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:10.038760 8002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:10.042097 8002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:10.042299 8002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:10.042891 8002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:10.043711 8002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:10.043790 8002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.043882 8002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:10.043932 8002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.052573 8002 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:34863
I20260924 20:14:10.052659 8130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:34863 every 8 connection(s)
I20260924 20:14:10.053493 8002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:14:10.055677 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 8002
I20260924 20:14:10.055814 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:14:10.056093 30030 external_mini_cluster.cc:1428] Setting key a1800890d917b6aa1c1ff9d70fce3f23
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[7761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280850, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:14:10.068964 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.055378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:50947 (local address 127.29.83.190:46113)
0924 20:14:10.055527 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.055532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.056541 (+ 1009us) server_negotiation.cc:407] Connection header received
0924 20:14:10.058166 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.058170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.058224 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.058325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:10.059732 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.060263 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.062074 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.062297 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.062381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.065207 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:10.065211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:10.065213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:14:10.065236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:10.066847 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.067396 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.067399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.067400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.067480 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.067838 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.067841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.067842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.068024 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:10.068104 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.068672 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.068797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:14:10.069913 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.055845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46113 (local address 127.29.83.130:50947)
0924 20:14:10.056372 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.056425 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.057956 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.058503 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.058510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.058866 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:10.059574 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.059595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.060399 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.060406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.061952 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.061967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.062463 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.062470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.062702 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.063336 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:10.063364 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:10.065035 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:10.067005 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:10.067023 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:10.067043 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:10.067297 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:10.067568 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:10.067571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:10.067572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:10.067757 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:10.068143 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:10.068149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:10.068430 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.068629 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.068959 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":163,"threads_started":1}
I20260924 20:14:10.071501 8131 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46113
I20260924 20:14:10.071782 8131 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:10.072327 8131 heartbeater.cc:511] Master 127.29.83.190:46113 requested a full tablet report, sending...
I20260924 20:14:10.073587 7806 ts_manager.cc:194] Registered new tserver with Master: 55bec0a615ea44f192222db892f2bbee (127.29.83.130:34863)
I20260924 20:14:10.074558 7806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:50947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:14:10.086549 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:14:10.201105 8136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:14:10.201470 8136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:14:10.201541 8136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:14:10.205667 8136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:14:10.205734 8136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:14:10.205858 8136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:14:10.211174 8136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:14:10.212208 8136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:14:10.213497 8136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:14:10.224191 8142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:10.224215 8144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:14:10.224195 8141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:14:10.224874 8136 server_base.cc:1034] running on GCE node
I20260924 20:14:10.225333 8136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:14:10.226138 8136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:14:10.227447 8136 hybrid_clock.cc:648] HybridClock initialized: now 1790280850227391 us; error 31 us; skew 500 ppm
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[7761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280850, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:14:10.236737 8136 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:14:10.237838 8136 webserver.cc:460] Webserver started at http://127.29.83.131:45793/ using document root <none> and password file <none>
I20260924 20:14:10.238440 8136 fs_manager.cc:359] Metadata directory not provided
I20260924 20:14:10.238535 8136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:14:10.238786 8136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:14:10.240872 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "c01f85eca5da4b75a229846120b0ce2f"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "b943a5cfb700e9bbe044020373d5a44a"
server_key_iv: "39279f0ba48fa78df36a0e886176216c"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.241446 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "c01f85eca5da4b75a229846120b0ce2f"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "b943a5cfb700e9bbe044020373d5a44a"
server_key_iv: "39279f0ba48fa78df36a0e886176216c"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.245355 8136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:14:10.247859 8151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.249061 8136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:14:10.249225 8136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "c01f85eca5da4b75a229846120b0ce2f"
format_stamp: "Formatted at 2026-09-24 20:14:10 on dist-test-slave-52fr"
server_key: "b943a5cfb700e9bbe044020373d5a44a"
server_key_iv: "39279f0ba48fa78df36a0e886176216c"
server_key_version: "encryptionkey@0"
I20260924 20:14:10.249338 8136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:14:10.273283 8136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:14:10.276178 8136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:14:10.276392 8136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:14:10.276968 8136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:14:10.277904 8136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:14:10.277956 8136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.277999 8136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:14:10.278049 8136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:14:10.286792 8136 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:41915
I20260924 20:14:10.286815 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:41915 every 8 connection(s)
I20260924 20:14:10.287706 8136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:14:10.293812 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 8136
I20260924 20:14:10.293965 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:14:10.294261 30030 external_mini_cluster.cc:1428] Setting key 93698fe59d2ac391ca6e282959ff8e60
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[7761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280850, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:14:10.301728 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.289610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:37251 (local address 127.29.83.190:46113)
0924 20:14:10.289809 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.289844 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.290661 (+ 817us) server_negotiation.cc:407] Connection header received
0924 20:14:10.292055 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.292059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.292118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.292222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:10.293381 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.294057 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.295448 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.295787 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.295873 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.298548 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:10.298551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:10.298553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:14:10.298578 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:10.299973 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.300439 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.300442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.300443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.300515 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.300796 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.300798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.300799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.300971 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:10.301048 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.301498 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.301598 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:14:10.302619 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.290040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:46113 (local address 127.29.83.131:37251)
0924 20:14:10.290482 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.290540 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.291833 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.292396 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.292403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.292765 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:14:10.293230 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.293243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.294184 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.294192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.295311 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.295320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.295936 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.295940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.296165 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.296833 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:14:10.296849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:14:10.298380 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:14:10.300083 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:10.300088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:10.300101 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:10.300353 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:10.300590 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:14:10.300592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:14:10.300594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:14:10.300719 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:14:10.301065 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:14:10.301070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:14:10.301298 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.301451 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.301711 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260924 20:14:10.303874 8265 heartbeater.cc:347] Connected to a master server at 127.29.83.190:46113
I20260924 20:14:10.304129 8265 heartbeater.cc:464] Registering TS with master...
I20260924 20:14:10.304636 8265 heartbeater.cc:511] Master 127.29.83.190:46113 requested a full tablet report, sending...
I20260924 20:14:10.305892 7806 ts_manager.cc:194] Registered new tserver with Master: c01f85eca5da4b75a229846120b0ce2f (127.29.83.131:41915)
I20260924 20:14:10.306547 7806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:37251
I20260924 20:14:10.308660 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:14:10.319329 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.310860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.29.83.190:46113)
0924 20:14:10.311040 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.311044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.311193 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:14:10.311362 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.311365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.311464 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.311554 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:14:10.312283 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.312833 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.313884 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.314140 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.314251 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.315681 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:14:10.315684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:14:10.315687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:14:10.315715 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:14:10.317165 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.317924 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.317928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.317930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.318038 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:14:10.318355 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:14:10.318358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:14:10.318360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:14:10.318606 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:14:10.318764 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.319055 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.319178 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:14:10.322634 7806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:14:10.325680 7806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:14:10.341667 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.335982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.29.83.129:45587)
0924 20:14:10.336896 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.336909 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.336927 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:14:10.336987 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.336990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.337127 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.337288 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:14:10.337513 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.338379 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.340305 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.340887 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341174 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.341376 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.341486 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":125,"threads_started":1}
I20260924 20:14:10.341681 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.335951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34863 (local address 127.0.0.1:58190)
0924 20:14:10.336385 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.336397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.336473 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.337392 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.337396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.337414 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:14:10.337495 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.337504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.339007 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.339010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.340021 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.340031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.341164 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.341267 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341291 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.341344 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.341502 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":5,"thread_start_us":103,"threads_started":1}
I20260924 20:14:10.341897 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.335220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.29.83.131:41915)
0924 20:14:10.335597 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.335610 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.336053 (+ 443us) server_negotiation.cc:407] Connection header received
0924 20:14:10.336300 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.336308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.336542 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.336672 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:14:10.337287 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.338494 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.340268 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.341014 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341134 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341280 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.341606 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.341706 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
I20260924 20:14:10.341661 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.335127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45587 (local address 127.0.0.1:49168)
0924 20:14:10.336211 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.336224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.336644 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.337263 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.337266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.337283 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:14:10.337386 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.337392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.338507 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.338512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.340210 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.340219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.341035 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.341154 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341205 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.341342 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.341506 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":113,"thread_start_us":89,"threads_started":1}
I20260924 20:14:10.342206 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.336505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.29.83.130:34863)
0924 20:14:10.336936 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.336949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.336969 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:14:10.337037 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.337041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.337244 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.337654 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:14:10.337827 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.338860 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.340152 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.341014 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341746 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341910 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.341961 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.342056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
I20260924 20:14:10.342492 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.335088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41915 (local address 127.0.0.1:53748)
0924 20:14:10.335883 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.335943 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.336106 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.336976 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.336983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.337008 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:14:10.337153 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.337159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.338628 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.338631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.340134 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.340145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.341339 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.341343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.341426 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.341477 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.342261 (+ 784us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.342343 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":145,"thread_start_us":105,"threads_started":1}
I20260924 20:14:10.344990 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 90bce67973774e988df5cfd4cf872543 (DEFAULT_TABLE table=test-table [id=c5ecc2560bef402ea0fd8b55e1e3241d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:14:10.345369 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 90bce67973774e988df5cfd4cf872543 (DEFAULT_TABLE table=test-table [id=c5ecc2560bef402ea0fd8b55e1e3241d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:14:10.345369 7931 tablet_service.cc:1467] Processing CreateTablet for tablet 90bce67973774e988df5cfd4cf872543 (DEFAULT_TABLE table=test-table [id=c5ecc2560bef402ea0fd8b55e1e3241d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:14:10.346194 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90bce67973774e988df5cfd4cf872543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:10.346452 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90bce67973774e988df5cfd4cf872543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:10.346447 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90bce67973774e988df5cfd4cf872543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:14:10.352491 8282 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Bootstrap starting.
I20260924 20:14:10.354121 8284 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Bootstrap starting.
I20260924 20:14:10.355302 8282 tablet_bootstrap.cc:654] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:10.355942 8284 tablet_bootstrap.cc:654] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:10.356096 8282 log.cc:824] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Log is configured to *not* fsync() on all Append() calls
I20260924 20:14:10.356845 8283 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Bootstrap starting.
I20260924 20:14:10.357172 8284 log.cc:824] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:14:10.358433 8282 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: No bootstrap required, opened a new log
I20260924 20:14:10.358664 8282 ts_tablet_manager.cc:1397] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:14:10.359258 8283 tablet_bootstrap.cc:654] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Neither blocks nor log segments found. Creating new log.
I20260924 20:14:10.359526 8284 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: No bootstrap required, opened a new log
I20260924 20:14:10.359706 8284 ts_tablet_manager.cc:1397] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:14:10.360090 8283 log.cc:824] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Log is configured to *not* fsync() on all Append() calls
I20260924 20:14:10.362203 8283 tablet_bootstrap.cc:492] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: No bootstrap required, opened a new log
I20260924 20:14:10.362129 8282 raft_consensus.cc:348] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.362406 8283 ts_tablet_manager.cc:1397] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:14:10.362425 8282 raft_consensus.cc:374] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:10.362478 8282 raft_consensus.cc:729] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55bec0a615ea44f192222db892f2bbee, State: Initialized, Role: FOLLOWER
I20260924 20:14:10.362985 8282 consensus_queue.cc:260] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.363659 8131 heartbeater.cc:503] Master 127.29.83.190:46113 was elected leader, sending a full tablet report...
I20260924 20:14:10.364037 8282 ts_tablet_manager.cc:1428] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:14:10.363543 8284 raft_consensus.cc:348] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.364492 8284 raft_consensus.cc:374] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:10.364545 8284 raft_consensus.cc:729] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01f85eca5da4b75a229846120b0ce2f, State: Initialized, Role: FOLLOWER
I20260924 20:14:10.365006 8284 consensus_queue.cc:260] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.365809 8265 heartbeater.cc:503] Master 127.29.83.190:46113 was elected leader, sending a full tablet report...
I20260924 20:14:10.366128 8284 ts_tablet_manager.cc:1428] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:14:10.370071 8283 raft_consensus.cc:348] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.370316 8283 raft_consensus.cc:374] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:14:10.370373 8283 raft_consensus.cc:729] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6459fe97944742f38c1b86fcbe6d1e94, State: Initialized, Role: FOLLOWER
I20260924 20:14:10.370806 8283 consensus_queue.cc:260] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.371510 7997 heartbeater.cc:503] Master 127.29.83.190:46113 was elected leader, sending a full tablet report...
I20260924 20:14:10.371968 8283 ts_tablet_manager.cc:1428] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260924 20:14:10.538853 8266 tablet.cc:2369] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:14:10.554790 8132 tablet.cc:2369] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:14:10.589681 7998 tablet.cc:2369] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:14:10.618913 8290 raft_consensus.cc:482] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:14:10.619138 8290 raft_consensus.cc:504] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.620306 8290 leader_election.cc:288] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c01f85eca5da4b75a229846120b0ce2f (127.29.83.131:41915), 55bec0a615ea44f192222db892f2bbee (127.29.83.130:34863)
I20260924 20:14:10.624541 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.620639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34087 (local address 127.29.83.131:41915)
0924 20:14:10.620798 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.620803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.621006 (+ 203us) server_negotiation.cc:407] Connection header received
0924 20:14:10.621178 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.621182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.621238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.621319 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:14:10.621722 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.622342 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.623488 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.623961 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.624057 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.624104 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.624322 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.624405 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:14:10.624555 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.620554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41915 (local address 127.29.83.129:34087)
0924 20:14:10.620886 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.620899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.621060 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.621345 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.621347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.621368 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:14:10.621571 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.621579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.622444 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.622447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.623349 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.623359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.624088 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.624092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.624195 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.624220 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.624293 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.624367 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"spinlock_wait_cycles":4480,"thread_start_us":132,"threads_started":1}
I20260924 20:14:10.624680 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.621108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:52645 (local address 127.29.83.130:34863)
0924 20:14:10.621243 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.621247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.621437 (+ 190us) server_negotiation.cc:407] Connection header received
0924 20:14:10.621567 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.621570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.621630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.621706 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:14:10.622013 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.622581 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.623689 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.624107 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.624214 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.624268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.624476 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.624557 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:14:10.624887 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.620996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34863 (local address 127.29.83.129:52645)
0924 20:14:10.621336 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.621349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:14:10.621453 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:14:10.621727 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:14:10.621730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:14:10.621747 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:14:10.621877 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.621883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.622704 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.622707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.623559 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:14:10.623569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.624226 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.624229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:14:10.624346 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.624368 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:14:10.624441 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:14:10.624502 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260924 20:14:10.625305 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90bce67973774e988df5cfd4cf872543" candidate_uuid: "6459fe97944742f38c1b86fcbe6d1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01f85eca5da4b75a229846120b0ce2f" is_pre_election: true
I20260924 20:14:10.625401 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90bce67973774e988df5cfd4cf872543" candidate_uuid: "6459fe97944742f38c1b86fcbe6d1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55bec0a615ea44f192222db892f2bbee" is_pre_election: true
I20260924 20:14:10.625633 8219 raft_consensus.cc:2454] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6459fe97944742f38c1b86fcbe6d1e94 in term 0.
I20260924 20:14:10.625690 8085 raft_consensus.cc:2454] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6459fe97944742f38c1b86fcbe6d1e94 in term 0.
I20260924 20:14:10.626151 7886 leader_election.cc:302] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6459fe97944742f38c1b86fcbe6d1e94, c01f85eca5da4b75a229846120b0ce2f; no voters:
I20260924 20:14:10.626441 8290 raft_consensus.cc:2781] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:14:10.626578 8290 raft_consensus.cc:482] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:14:10.626646 8290 raft_consensus.cc:3037] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:10.627732 8290 raft_consensus.cc:504] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.628140 8290 leader_election.cc:288] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [CANDIDATE]: Term 1 election: Requested vote from peers c01f85eca5da4b75a229846120b0ce2f (127.29.83.131:41915), 55bec0a615ea44f192222db892f2bbee (127.29.83.130:34863)
I20260924 20:14:10.628504 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90bce67973774e988df5cfd4cf872543" candidate_uuid: "6459fe97944742f38c1b86fcbe6d1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01f85eca5da4b75a229846120b0ce2f"
I20260924 20:14:10.628639 8219 raft_consensus.cc:3037] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:10.628809 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90bce67973774e988df5cfd4cf872543" candidate_uuid: "6459fe97944742f38c1b86fcbe6d1e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55bec0a615ea44f192222db892f2bbee"
I20260924 20:14:10.628933 8085 raft_consensus.cc:3037] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:14:10.629839 8219 raft_consensus.cc:2454] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6459fe97944742f38c1b86fcbe6d1e94 in term 1.
I20260924 20:14:10.630106 7886 leader_election.cc:302] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6459fe97944742f38c1b86fcbe6d1e94, c01f85eca5da4b75a229846120b0ce2f; no voters:
I20260924 20:14:10.630134 8085 raft_consensus.cc:2454] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6459fe97944742f38c1b86fcbe6d1e94 in term 1.
I20260924 20:14:10.630275 8290 raft_consensus.cc:2781] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:14:10.630463 8290 raft_consensus.cc:686] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [term 1 LEADER]: Becoming Leader. State: Replica: 6459fe97944742f38c1b86fcbe6d1e94, State: Running, Role: LEADER
I20260924 20:14:10.630831 8290 consensus_queue.cc:237] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } }
I20260924 20:14:10.634254 7806 catalog_manager.cc:5495] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6459fe97944742f38c1b86fcbe6d1e94 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "6459fe97944742f38c1b86fcbe6d1e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6459fe97944742f38c1b86fcbe6d1e94" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 } health_report { overall_health: UNKNOWN } } }
I20260924 20:14:10.706372 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.702659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.29.83.129:45587)
0924 20:14:10.702823 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:14:10.702827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:14:10.702843 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:14:10.702932 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:14:10.702934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:14:10.703006 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:14:10.703104 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:14:10.703340 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.703915 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.704824 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:14:10.705056 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:14:10.705150 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:14:10.705453 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:14:10.705995 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:14:10.706101 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:14:10.706199 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 20:14:10.706255 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:14:10.712605 8219 raft_consensus.cc:1261] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 1 FOLLOWER]: Refusing update from remote peer 6459fe97944742f38c1b86fcbe6d1e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:14:10.712600 8085 raft_consensus.cc:1261] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 1 FOLLOWER]: Refusing update from remote peer 6459fe97944742f38c1b86fcbe6d1e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:14:10.713418 8290 consensus_queue.cc:1035] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01f85eca5da4b75a229846120b0ce2f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:14:10.713641 8293 consensus_queue.cc:1035] T 90bce67973774e988df5cfd4cf872543 P 6459fe97944742f38c1b86fcbe6d1e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55bec0a615ea44f192222db892f2bbee" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:14:10.719579 8296 mvcc.cc:205] Tried to move back new op lower bound from 7332990364511449088 to 7332990364186476544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990364511449088}]
I20260924 20:14:10.720548 8298 mvcc.cc:205] Tried to move back new op lower bound from 7332990364511449088 to 7332990364186476544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990364511449088}]
I20260924 20:14:10.723246 8297 mvcc.cc:205] Tried to move back new op lower bound from 7332990364511449088 to 7332990364186476544. Current Snapshot: MvccSnapshot[applied={T|T < 7332990364511449088}]
I20260924 20:14:10.729511 7805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:14:10.729688 7805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:14:10.732007 7805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce2fd0c9ba3b4604b7bfc2a7c286a86b: Sending DeleteTablet for 3 replicas of tablet 90bce67973774e988df5cfd4cf872543
I20260924 20:14:10.732795 8065 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bce67973774e988df5cfd4cf872543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:14:10 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:58190
I20260924 20:14:10.732795 7931 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bce67973774e988df5cfd4cf872543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:14:10 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:49168
I20260924 20:14:10.732878 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bce67973774e988df5cfd4cf872543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:14:10 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53748
I20260924 20:14:10.733253 8306 tablet_replica.cc:330] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: stopping tablet replica
I20260924 20:14:10.733443 8308 tablet_replica.cc:330] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: stopping tablet replica
I20260924 20:14:10.733578 8306 raft_consensus.cc:2229] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:14:10.733844 8308 raft_consensus.cc:2229] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:14:10.733946 8306 raft_consensus.cc:2258] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:14:10.734014 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7868
I20260924 20:14:10.734249 8308 raft_consensus.cc:2258] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:14:10.736007 8306 ts_tablet_manager.cc:1905] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:14:10.737952 8308 ts_tablet_manager.cc:1905] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:14:10.738672 8306 ts_tablet_manager.cc:1918] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:14:10.738772 8306 log.cc:1196] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/90bce67973774e988df5cfd4cf872543
I20260924 20:14:10.739089 8306 ts_tablet_manager.cc:1939] T 90bce67973774e988df5cfd4cf872543 P 55bec0a615ea44f192222db892f2bbee: Deleting consensus metadata
I20260924 20:14:10.740702 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 8002
W20260924 20:14:10.740670 7790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:45587 (error 108)
I20260924 20:14:10.740684 7792 catalog_manager.cc:4841] TS 55bec0a615ea44f192222db892f2bbee (127.29.83.130:34863): tablet 90bce67973774e988df5cfd4cf872543 (table test-table [id=c5ecc2560bef402ea0fd8b55e1e3241d]) successfully deleted
I20260924 20:14:10.742206 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:14:10.741839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45587 (local address 127.0.0.1:49184)
0924 20:14:10.742043 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 20:14:10.742123 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 20:14:10.742362 7790 catalog_manager.cc:4584] TS 6459fe97944742f38c1b86fcbe6d1e94 (127.29.83.129:45587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90bce67973774e988df5cfd4cf872543: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45587: connect: Connection refused (error 111)
I20260924 20:14:10.746346 8308 ts_tablet_manager.cc:1918] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:14:10.746459 8308 log.cc:1196] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/90bce67973774e988df5cfd4cf872543
I20260924 20:14:10.746587 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 8136
I20260924 20:14:10.746855 8308 ts_tablet_manager.cc:1939] T 90bce67973774e988df5cfd4cf872543 P c01f85eca5da4b75a229846120b0ce2f: Deleting consensus metadata
I20260924 20:14:10.752137 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 7774
2026-09-24T20:14:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2554 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[8312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[8312](info): set up 2 sockets
Sep 24 20:14:10 dist-test-slave-52fr krb5kdc[8312](info): commencing operation
krb5kdc: starting...
W20260924 20:14:11.806797 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:14:11 dist-test-slave-52fr krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:14:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/rangeradmin_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/rangeradmin_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/rangerlookup_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/rangerlookup_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
I20260924 20:14:11.980551 30030 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrGCYXn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/postgres -l logfile start
2026-09-24 20:14:14.690 UTC [8342] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:14:14.690 UTC [8342] LOG: listening on IPv4 address "127.29.83.148", port 35539
2026-09-24 20:14:14.694 UTC [8342] LOG: listening on Unix socket "/tmp/.s.PGSQL.35539"
2026-09-24 20:14:14.716 UTC [8345] LOG: database system was shut down at 2026-09-24 20:14:13 UTC
2026-09-24 20:14:14.719 UTC [8342] LOG: database system is ready to accept connections
I20260924 20:14:15.678432 30030 mini_postgres.cc:96] Postgres bound to 35539
2026-09-24 20:14:15.684 UTC [8353] FATAL: database "slave" does not exist
127.29.83.148:35539 - accepting connections
I20260924 20:14:15.685256 30030 mini_ranger.cc:161] Starting Ranger...
I20260924 20:14:15.701207 30030 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:14:16.249867 30030 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:14:16.249969 30030 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin
2026-09-24 20:14:16,513 [I] DB FLAVOR :POSTGRES
2026-09-24 20:14:16,514 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:14:16,514 [I] Checking connection..
2026-09-24 20:14:16,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:14:16,776 [I] Checking connection passed.
2026-09-24 20:14:16,776 [I] --------- Verifying version history table ---------
2026-09-24 20:14:16,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:14:17,036 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:14:17,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:14:17,266 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:14:17,266 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:14:17,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:14:17.503 UTC [8417] WARNING: there is no transaction in progress
2026-09-24 20:14:17,509 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:14:17,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:14:17,760 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:14:17,760 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:14:17,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:14:18,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:14:18,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-52fr.c.gcp-upstream.internal','N') ;"
2026-09-24 20:14:18,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:14:18,754 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:14:18,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:14:18,995 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:14:18,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:14:19,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:14:19,481 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:14:19,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:14:20.482 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.500 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.518 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.534 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.553 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.574 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.634 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.641 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.656 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.666 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.677 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.690 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.700 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.707 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.713 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:20.720 UTC [8525] WARNING: there is no transaction in progress
2026-09-24 20:14:21,403 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:14:21,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:14:21,653 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:14:21,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:14:21,890 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:14:21,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:14:22,139 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:14:22,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:14:22,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:14:22,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-52fr.c.gcp-upstream.internal';"
2026-09-24 20:14:22,877 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:14:22,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:35539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:14:23,119 [I] DB_PATCHES have already been applied
I20260924 20:14:23.124709 30030 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrGCYXn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:14:23.124806 30030 mini_ranger.cc:193] Using host: 127.29.83.148
I20260924 20:14:23.128304 30030 mini_ranger.cc:227] Ranger admin URL: http://127.29.83.148:44615
Sep 24, 2026 8:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:14:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:14:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 24, 2026 8:14:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:14:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 8:14:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:14:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:14:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:16:23.685496 30030 mini_ranger.cc:160] Time spent starting Ranger: real 128.000s user 0.000s sys 0.159s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 8612 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p8612
f310
n127.0.0.1:38666->127.29.83.148:35539
f311
n127.0.0.1:38646->127.29.83.148:35539
f312
n127.0.0.1:38648->127.29.83.148:35539
f313
n127.0.0.1:38670->127.29.83.148:35539
f314
n127.0.0.1:38668->127.29.83.148:35539'
I20260924 20:16:23.686499 30030 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:16:23.717409 30030 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:16:23.717 UTC [8342] LOG: received smart shutdown request
2026-09-24 20:16:23.721 UTC [8342] LOG: background worker "logical replication launcher" (PID 8351) exited with exit code 1
2026-09-24 20:16:23.721 UTC [8346] LOG: shutting down
2026-09-24 20:16:23.899 UTC [8342] LOG: database system is shut down
2026-09-24T20:16:23Z chronyd exiting
I20260924 20:16:23.911531 30030 test_util.cc:175] -----------------------------------------------
I20260924 20:16:23.911593 30030 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280784619584-30030-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (133138 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:16:23 dist-test-slave-52fr krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:16:23 dist-test-slave-52fr krb5kdc[8733](info): set up 2 sockets
Sep 24 20:16:23 dist-test-slave-52fr krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260924 20:16:24.933434 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:16:25 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:16:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangeradmin_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangeradmin_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangerlookup_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangerlookup_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
I20260924 20:16:25.112777 30030 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrGCYXn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/postgres -l logfile start
2026-09-24 20:16:27.732 UTC [8763] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:16:27.732 UTC [8763] LOG: listening on IPv4 address "127.29.83.148", port 49109
2026-09-24 20:16:27.736 UTC [8763] LOG: listening on Unix socket "/tmp/.s.PGSQL.49109"
2026-09-24 20:16:27.757 UTC [8766] LOG: database system was shut down at 2026-09-24 20:16:26 UTC
2026-09-24 20:16:27.760 UTC [8763] LOG: database system is ready to accept connections
I20260924 20:16:28.713301 30030 mini_postgres.cc:96] Postgres bound to 49109
2026-09-24 20:16:28.719 UTC [8774] FATAL: database "slave" does not exist
127.29.83.148:49109 - accepting connections
I20260924 20:16:28.720309 30030 mini_ranger.cc:161] Starting Ranger...
I20260924 20:16:28.734825 30030 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:16:29.236981 30030 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:16:29.237074 30030 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin
2026-09-24 20:16:29,512 [I] DB FLAVOR :POSTGRES
2026-09-24 20:16:29,513 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:16:29,513 [I] Checking connection..
2026-09-24 20:16:29,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:16:29,790 [I] Checking connection passed.
2026-09-24 20:16:29,790 [I] --------- Verifying version history table ---------
2026-09-24 20:16:29,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:16:30,022 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:16:30,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:16:30,288 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:16:30,289 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:16:30,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:16:30.547 UTC [8838] WARNING: there is no transaction in progress
2026-09-24 20:16:30,554 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:16:30,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:16:30,782 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:16:30,783 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:16:30,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:16:31,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:16:31,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-52fr.c.gcp-upstream.internal','N') ;"
2026-09-24 20:16:31,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:16:31,756 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:16:31,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:16:32,026 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:16:32,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:16:32,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:16:32,522 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:16:32,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:16:33.531 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.547 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.564 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.581 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.601 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.622 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.683 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.690 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.705 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.717 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.728 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.741 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.751 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.757 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.763 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:33.770 UTC [8946] WARNING: there is no transaction in progress
2026-09-24 20:16:34,474 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:16:34,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:16:34,697 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:16:34,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:16:34,964 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:16:34,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:16:35,211 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:16:35,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:16:35,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:16:35,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-52fr.c.gcp-upstream.internal';"
2026-09-24 20:16:35,934 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:16:35,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:16:36,180 [I] DB_PATCHES have already been applied
I20260924 20:16:36.185701 30030 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrGCYXn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:16:36.185825 30030 mini_ranger.cc:193] Using host: 127.29.83.148
I20260924 20:16:36.189211 30030 mini_ranger.cc:227] Ranger admin URL: http://127.29.83.148:42163
Sep 24, 2026 8:16:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:16:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:16:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:16:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:16:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 20:16:37.307044 30030 mini_ranger.cc:160] Time spent starting Ranger: real 8.587s user 0.000s sys 0.010s
Sep 24, 2026 8:16:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:16:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 8:16:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:16:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:16:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:16:58.992041 30030 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.83.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.83.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangerkms_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.83.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/rangerkms_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.83.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.83.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.83.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/HTTP_127.29.83.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 20:16:59 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281019, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 20:16:59.094187 30030 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 20:16:59.104707 30030 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 20:17:00.056241 30030 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 20:17:00.056347 30030 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms
I20260924 20:17:00.056465 30030 mini_ranger_kms.cc:227] Using postgres at 127.29.83.148:49109
2026-09-24 20:17:00,250 [I] DB FLAVOR :POSTGRES
2026-09-24 20:17:00,251 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:17:00,251 [I] Checking connection
2026-09-24 20:17:00,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 20:17:00,503 [I] connection success
2026-09-24 20:17:00,504 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 20:17:00,504 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 20:17:00,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 20:17:00,758 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 20:17:00,759 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:17:00,759 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 20:17:00,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.83.148:49109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 20:17:01,035 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 20:17:01.548398 30030 mini_ranger_kms.cc:310] Created kms service
I20260924 20:17:01.764213 30030 mini_ranger_kms.cc:326] Created kudu user
I20260924 20:17:01.813328 30030 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 20:17:01 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281019, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:02.230826 30030 mini_ranger_kms.cc:384] Added ranger policy
I20260924 20:17:02.230986 30030 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrGCYXn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrGCYXn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 20:17:02.231016 30030 mini_ranger_kms.cc:240] Using host: 127.29.83.148
I20260924 20:17:02.235185 30030 mini_ranger_kms.cc:276] Ranger KMS PID: 9126
I20260924 20:17:02.235338 30030 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.83.148:48923
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 8:17:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 20:17:04.557358 30030 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.463s user 0.006s sys 0.172s
I20260924 20:17:04.557508 30030 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 20:17:04.557631 30030 mini_ranger_kms.cc:401] 127.29.83.148:48923/kms/v1/keys
Sep 24 20:17:04 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.83.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 20:17:05 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.83.148@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
Sep 24 20:17:05 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.83.148@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
Sep 24 20:17:07 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:17:07.857635 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36907
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36907
--ranger_config_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:08.011289 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:08.011725 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:08.011791 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:08.017570 9177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:08.017652 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:08.017680 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:08.017704 9177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:08.017725 9177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:08.025549 9177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.83.148:48923/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36907
--ranger_config_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:08.027068 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:08.028473 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:08.038993 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:08.039005 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:08.039286 9177 server_base.cc:1034] running on GCE node
W20260924 20:17:08.039341 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:08.040071 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:08.041566 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:08.042773 9177 hybrid_clock.cc:648] HybridClock initialized: now 1790281028042770 us; error 52 us; skew 500 ppm
Sep 24 20:17:08 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281028, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:08.046365 9177 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:08.047463 9177 webserver.cc:460] Webserver started at http://127.29.83.190:43511/ using document root <none> and password file <none>
I20260924 20:17:08.048110 9177 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:08.048200 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:08.048460 9177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:17:08 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281028, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:08.104285 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "4a121b25e39f442b9029d36c74b44cf9"
format_stamp: "Formatted at 2026-09-24 20:17:08 on dist-test-slave-52fr"
server_key: "a0d7543eea15273455294f068773fc17"
server_key_iv: "ac43e8d4fcf286878913467a2a0426d9"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:08.104948 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "4a121b25e39f442b9029d36c74b44cf9"
format_stamp: "Formatted at 2026-09-24 20:17:08 on dist-test-slave-52fr"
server_key: "a0d7543eea15273455294f068773fc17"
server_key_iv: "ac43e8d4fcf286878913467a2a0426d9"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:08.109145 9177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:17:08.123697 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:08.125200 9177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:08.125341 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "4a121b25e39f442b9029d36c74b44cf9"
format_stamp: "Formatted at 2026-09-24 20:17:08 on dist-test-slave-52fr"
server_key: "a0d7543eea15273455294f068773fc17"
server_key_iv: "ac43e8d4fcf286878913467a2a0426d9"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:08.125484 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:08.141927 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:08.145368 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:08.145605 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:08.154493 9177 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36907
I20260924 20:17:08.154531 9244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36907 every 8 connection(s)
I20260924 20:17:08.155556 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:08.157186 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9177
I20260924 20:17:08.157331 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:17:08.157696 30030 external_mini_cluster.cc:1428] Setting key 8afd7e14c03f0d1e7f03652cad59d63d
I20260924 20:17:08.159222 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:08.166524 9245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Bootstrap starting.
Sep 24 20:17:08 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:08.169921 9245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:08.171303 9245 log.cc:824] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:08.172602 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:08.159293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.29.83.190:36907)
0924 20:17:08.159745 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:08.159766 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:08.159815 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:17:08.160520 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:08.160537 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:08.160872 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:08.161234 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:08.162215 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:08.163363 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:08.164272 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:08.164750 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:08.164940 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:08.167714 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:08.167719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:08.167733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:08.167764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:08.169508 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:08.170132 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:08.170137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:08.170142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:08.170250 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:08.170568 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:08.170576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:08.170578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:08.170986 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:08.171193 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:08.171477 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 20:17:08.171785 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260924 20:17:08.173800 9245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: No bootstrap required, opened a new log
I20260924 20:17:08.176893 9245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:08.177150 9245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:08.177229 9245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a121b25e39f442b9029d36c74b44cf9, State: Initialized, Role: FOLLOWER
I20260924 20:17:08.177686 9245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:08.178064 9245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:08.178181 9245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:08.178297 9245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:08.179507 9245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:08.179895 9245 leader_election.cc:302] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a121b25e39f442b9029d36c74b44cf9; no voters:
I20260924 20:17:08.180335 9245 leader_election.cc:288] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:17:08.180459 9250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:08.180680 9250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a121b25e39f442b9029d36c74b44cf9, State: Running, Role: LEADER
I20260924 20:17:08.181069 9250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:08.181794 9245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:08.182569 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a121b25e39f442b9029d36c74b44cf9. Latest consensus state: current_term: 1 leader_uuid: "4a121b25e39f442b9029d36c74b44cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } } }
I20260924 20:17:08.183286 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:08.183184 9251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a121b25e39f442b9029d36c74b44cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } } }
I20260924 20:17:08.183573 9251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:08.183863 9260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:08.184142 9245 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:17:08.192399 9260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:08.200632 9260 catalog_manager.cc:1347] Generated new cluster ID: 344850b1b6794865813882681898c938
I20260924 20:17:08.200708 9260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:08.219271 9260 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:17:08.220757 9260 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:08.232102 9260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Generated new TSK 0
I20260924 20:17:08.233170 9260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:17:09.012316 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:09.185425 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:09.185866 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:09.186012 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:09.192653 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:09.192740 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:09.192857 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:09.201086 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:09.203096 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:09.204677 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:09.217914 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.217948 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.217924 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:09.218588 9290 server_base.cc:1034] running on GCE node
I20260924 20:17:09.219115 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:09.219980 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:09.223330 9290 hybrid_clock.cc:648] HybridClock initialized: now 1790281029223282 us; error 50 us; skew 500 ppm
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:09.227588 9290 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:09.228734 9290 webserver.cc:460] Webserver started at http://127.29.83.129:41505/ using document root <none> and password file <none>
I20260924 20:17:09.229415 9290 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:09.229516 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:09.229857 9290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:09.254710 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "0502eef23347409092aff514f2d06fe7"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "ce95a7f5f07303932f3c79b956896119"
server_key_iv: "91bbcb88ec32a1ad28d9afeeafa34cea"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.256884 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "0502eef23347409092aff514f2d06fe7"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "ce95a7f5f07303932f3c79b956896119"
server_key_iv: "91bbcb88ec32a1ad28d9afeeafa34cea"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.271631 9290 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260924 20:17:09.301311 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.303458 9290 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260924 20:17:09.303678 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "0502eef23347409092aff514f2d06fe7"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "ce95a7f5f07303932f3c79b956896119"
server_key_iv: "91bbcb88ec32a1ad28d9afeeafa34cea"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.303889 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:09.329108 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:09.333448 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:09.333747 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:09.334610 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:09.335740 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:09.335841 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.335968 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:09.336205 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.356343 9290 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:34033
I20260924 20:17:09.356376 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:34033 every 8 connection(s)
I20260924 20:17:09.357558 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:09.363240 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9290
I20260924 20:17:09.363366 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:17:09.363722 30030 external_mini_cluster.cc:1428] Setting key e4bf8ddfda5929b9051653937ca34b33
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:17:09.377687 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:09.360267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.129:34517)
0924 20:17:09.360775 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:09.360835 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:09.361855 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:09.362497 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:09.362505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:09.362845 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:09.363585 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:09.363608 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.364718 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.364722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:09.366135 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:09.366148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.366998 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.367006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:09.367280 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:09.368103 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:09.368144 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:09.370101 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:09.373752 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:09.373796 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:09.373820 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:09.374312 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:09.374706 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:09.374710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:09.374712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:09.374937 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:09.375490 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:09.375498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:09.375880 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 20:17:09.376113 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 20:17:09.376544 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20260924 20:17:09.378582 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:09.359792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34517 (local address 127.29.83.190:36907)
0924 20:17:09.361346 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:09.361352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:09.361373 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:17:09.362097 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:09.362101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:09.362168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:09.362303 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:09.363784 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.364556 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.366294 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.366655 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.366779 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:09.370375 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:09.370382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:09.370388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:09.370427 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:09.373550 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:09.374484 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:09.374488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:09.374490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:09.374591 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:09.375051 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:09.375055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:09.375057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:09.375314 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:09.375432 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:09.378196 (+ 2764us) server_negotiation.cc:299] Negotiation successful
0924 20:17:09.378352 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":73,"threads_started":1}
I20260924 20:17:09.379786 9423 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:17:09.380061 9423 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:09.380602 9423 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:09.382427 9209 ts_manager.cc:194] Registered new tserver with Master: 0502eef23347409092aff514f2d06fe7 (127.29.83.129:34033)
I20260924 20:17:09.383704 9209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:34517
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:17:09.398536 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:09.528406 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:09.528699 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:09.528771 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:09.533246 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:09.533324 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:09.533416 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:09.540002 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:09.541168 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:09.542263 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:09.553824 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.553911 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.554412 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:09.555147 9429 server_base.cc:1034] running on GCE node
I20260924 20:17:09.555641 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:09.556352 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:09.557547 9429 hybrid_clock.cc:648] HybridClock initialized: now 1790281029557508 us; error 47 us; skew 500 ppm
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:09.561048 9429 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:09.562160 9429 webserver.cc:460] Webserver started at http://127.29.83.130:41417/ using document root <none> and password file <none>
I20260924 20:17:09.562748 9429 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:09.562840 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:09.563102 9429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:09.581182 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "d8d40a71cf2c4b93b411abb081a26711"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "47f0fbe89a4b8331abeab47f435a7cdd"
server_key_iv: "014a2834cff594c7e8a4375b35ef78f0"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.582175 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d40a71cf2c4b93b411abb081a26711"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "47f0fbe89a4b8331abeab47f435a7cdd"
server_key_iv: "014a2834cff594c7e8a4375b35ef78f0"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.587028 9429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:17:09.606247 9445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.608242 9429 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260924 20:17:09.608656 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "d8d40a71cf2c4b93b411abb081a26711"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "47f0fbe89a4b8331abeab47f435a7cdd"
server_key_iv: "014a2834cff594c7e8a4375b35ef78f0"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.608919 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:09.633091 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:09.637115 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:09.637393 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:09.638166 9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:09.639364 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:09.639472 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.639570 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:09.639632 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:09.653978 9429 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:33595
I20260924 20:17:09.655262 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:09.656915 9558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:33595 every 8 connection(s)
I20260924 20:17:09.659497 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9429
I20260924 20:17:09.659611 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:17:09.659935 30030 external_mini_cluster.cc:1428] Setting key 6ddad1c2b061a91b81c09e55697056f7
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:17:09.684497 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:09.659074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:60075 (local address 127.29.83.190:36907)
0924 20:17:09.659218 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:09.659222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:09.665400 (+ 6178us) server_negotiation.cc:407] Connection header received
0924 20:17:09.666751 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:09.666756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:09.666813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:09.666867 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:09.668684 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.669500 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.675598 (+ 6098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.675981 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.676084 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:09.679509 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:09.679517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:09.679520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:09.679556 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:09.681530 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:09.682393 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:09.682398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:09.682400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:09.682499 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:09.682975 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:09.682978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:09.682980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:09.683221 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:09.683309 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:09.684138 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 20:17:09.684293 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:17:09.686318 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:09.659607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.130:60075)
0924 20:17:09.665198 (+ 5591us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:09.665272 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:09.666469 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:09.667342 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:09.667352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:09.667724 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:09.668490 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:09.668522 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.669668 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.669672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:09.671155 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:09.671165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:09.676342 (+ 5177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:09.676346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:09.676580 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:09.677359 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:09.677385 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:09.679234 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:09.681702 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:09.681711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:09.681734 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:09.682188 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:09.682625 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:09.682628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:09.682630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:09.682853 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:09.683429 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:09.683437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:09.683817 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 20:17:09.684746 (+ 929us) client_negotiation.cc:240] Negotiation successful
0924 20:17:09.685134 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5305,"thread_start_us":147,"threads_started":1}
I20260924 20:17:09.687738 9559 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
I20260924 20:17:09.688004 9559 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:09.688625 9559 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:09.689802 9209 ts_manager.cc:194] Registered new tserver with Master: d8d40a71cf2c4b93b411abb081a26711 (127.29.83.130:33595)
I20260924 20:17:09.690516 9209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:60075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:17:09.710415 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:09.880406 9564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:09.881258 9564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:09.881418 9564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:09.887981 9564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:09.888134 9564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:09.888270 9564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:09.896660 9564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:09.898453 9564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:09.899981 9564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:09.916198 9572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.919245 9570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:09.919426 9569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:09.920810 9564 server_base.cc:1034] running on GCE node
I20260924 20:17:09.921556 9564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:09.922470 9564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:09.954046 9564 hybrid_clock.cc:648] HybridClock initialized: now 1790281029953908 us; error 166 us; skew 500 ppm
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:09.958810 9564 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:09.960134 9564 webserver.cc:460] Webserver started at http://127.29.83.131:39449/ using document root <none> and password file <none>
I20260924 20:17:09.960954 9564 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:09.961114 9564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:09.961417 9564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:17:09 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:09.991546 9564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "8e5ffe38913d43ba862828b725f6928f"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "9c981145a32a4415ae907d8634ac579b"
server_key_iv: "c413d2676f84b8646e622c40dfdbae9c"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:09.992558 9564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "8e5ffe38913d43ba862828b725f6928f"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "9c981145a32a4415ae907d8634ac579b"
server_key_iv: "c413d2676f84b8646e622c40dfdbae9c"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:10.007788 9564 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260924 20:17:10.048412 9579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.050396 9564 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.002s sys 0.000s
I20260924 20:17:10.050559 9564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "8e5ffe38913d43ba862828b725f6928f"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "9c981145a32a4415ae907d8634ac579b"
server_key_iv: "c413d2676f84b8646e622c40dfdbae9c"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:10.050717 9564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:10.076951 9564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:10.081076 9564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:10.081352 9564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:10.082231 9564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:10.083436 9564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:10.083549 9564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.083649 9564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:10.083714 9564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.095180 9564 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42833
I20260924 20:17:10.096572 9564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:10.097608 9692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42833 every 8 connection(s)
I20260924 20:17:10.104843 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9564
I20260924 20:17:10.104979 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:17:10.105338 30030 external_mini_cluster.cc:1428] Setting key b6b23b6f89006e3f84ba57ac1e867db1
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281029, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:10.123713 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.109474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:58235 (local address 127.29.83.190:36907)
0924 20:17:10.109605 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:10.109610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:10.110428 (+ 818us) server_negotiation.cc:407] Connection header received
0924 20:17:10.111645 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:10.111649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:10.111702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:10.111787 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:10.113392 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.114269 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.116186 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.116541 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.116629 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:10.119878 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:10.119881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:10.119885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:10.119919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:10.121490 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:10.122152 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:10.122155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:10.122156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:10.122227 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:10.122602 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:10.122605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:10.122607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:10.122791 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:10.122896 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:10.123426 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 20:17:10.123553 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:17:10.124840 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.108269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.131:58235)
0924 20:17:10.110249 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:10.110312 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:10.111388 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:10.112120 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:10.112130 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:10.112492 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:10.113188 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:10.113214 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.114471 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.114477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:10.116041 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:10.116053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.116905 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.116909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:10.117114 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:10.117880 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:10.117901 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:10.119682 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:10.121615 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:10.121622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:10.121641 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:10.121996 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:10.122325 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:10.122332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:10.122336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:10.122499 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:10.122906 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:10.122915 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:10.123183 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 20:17:10.123377 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 20:17:10.123709 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":173,"threads_started":1}
I20260924 20:17:10.126078 9693 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
I20260924 20:17:10.126314 9693 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:10.126801 9693 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:10.127806 9209 ts_manager.cc:194] Registered new tserver with Master: 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833)
I20260924 20:17:10.128405 9209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:58235
I20260924 20:17:10.131712 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:17:10.132359 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9290
I20260924 20:17:10.138561 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9429
I20260924 20:17:10.144793 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9564
I20260924 20:17:10.151347 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9177
I20260924 20:17:10.159619 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36907
--webserver_interface=127.29.83.190
--webserver_port=43511
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36907
--ranger_config_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:10.275982 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:10.276275 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:10.276319 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:10.280391 9696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:10.280467 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:10.280488 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:10.280504 9696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:10.280520 9696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:10.286342 9696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36907
--ranger_config_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:10.287429 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:10.288450 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:10.297704 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.297602 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.297683 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:10.298218 9696 server_base.cc:1034] running on GCE node
I20260924 20:17:10.298846 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:10.300079 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:10.301323 9696 hybrid_clock.cc:648] HybridClock initialized: now 1790281030301292 us; error 51 us; skew 500 ppm
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:10.304692 9696 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:10.305675 9696 webserver.cc:460] Webserver started at http://127.29.83.190:43511/ using document root <none> and password file <none>
I20260924 20:17:10.306311 9696 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:10.306416 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:10.376931 9696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:10.379737 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.381276 9696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:10.381461 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "4a121b25e39f442b9029d36c74b44cf9"
format_stamp: "Formatted at 2026-09-24 20:17:08 on dist-test-slave-52fr"
server_key: "a0d7543eea15273455294f068773fc17"
server_key_iv: "ac43e8d4fcf286878913467a2a0426d9"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:10.382110 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:10.392689 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:10.395745 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:10.396210 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:10.403548 9764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36907 every 8 connection(s)
I20260924 20:17:10.403575 9696 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36907
I20260924 20:17:10.404693 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:10.407963 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9696
I20260924 20:17:10.408597 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:34033
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:36907
--webserver_port=41505
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:10.410449 9765 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:17:10.413182 9765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Bootstrap starting.
I20260924 20:17:10.431010 9765 log.cc:824] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:10.438759 9765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:10.439193 9765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Bootstrap complete.
I20260924 20:17:10.443411 9765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:10.443851 9765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a121b25e39f442b9029d36c74b44cf9, State: Initialized, Role: FOLLOWER
I20260924 20:17:10.444365 9765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:10.444480 9765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:10.444568 9765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:10.444664 9765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:17:10.446713 9765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:10.447079 9765 leader_election.cc:302] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a121b25e39f442b9029d36c74b44cf9; no voters:
I20260924 20:17:10.447446 9765 leader_election.cc:288] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:17:10.447539 9769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:17:10.447779 9769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [term 2 LEADER]: Becoming Leader. State: Replica: 4a121b25e39f442b9029d36c74b44cf9, State: Running, Role: LEADER
I20260924 20:17:10.448088 9769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } }
I20260924 20:17:10.448764 9765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:10.449090 9769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a121b25e39f442b9029d36c74b44cf9. Latest consensus state: current_term: 2 leader_uuid: "4a121b25e39f442b9029d36c74b44cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } } }
I20260924 20:17:10.449210 9769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:10.449573 9773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:10.449615 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a121b25e39f442b9029d36c74b44cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a121b25e39f442b9029d36c74b44cf9" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36907 } } }
I20260924 20:17:10.449858 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:10.453034 9773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:10.453589 9765 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:17:10.453702 9773 catalog_manager.cc:1259] Loaded cluster ID: 344850b1b6794865813882681898c938
I20260924 20:17:10.453863 9773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:10.456576 9773 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:10.457355 9773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Loaded TSK: 0
I20260924 20:17:10.458662 9773 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:17:10.556449 9767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:10.556785 9767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:10.556865 9767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:10.561276 9767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:10.561383 9767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:10.561494 9767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:10.569269 9767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:34033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=41505
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:10.570770 9767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:10.571982 9767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:10.582427 9802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.582666 9805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.582474 9803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:10.582917 9767 server_base.cc:1034] running on GCE node
I20260924 20:17:10.583412 9767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:10.584180 9767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:10.585397 9767 hybrid_clock.cc:648] HybridClock initialized: now 1790281030585359 us; error 46 us; skew 500 ppm
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:10.589280 9767 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:10.590476 9767 webserver.cc:460] Webserver started at http://127.29.83.129:41505/ using document root <none> and password file <none>
I20260924 20:17:10.591104 9767 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:10.591221 9767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:10.617522 9767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:17:10.624150 9813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.627893 9767 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 20:17:10.628070 9767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "0502eef23347409092aff514f2d06fe7"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "ce95a7f5f07303932f3c79b956896119"
server_key_iv: "91bbcb88ec32a1ad28d9afeeafa34cea"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:10.628894 9767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:10.650513 9767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:10.654495 9767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:10.654716 9767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:10.655445 9767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:10.656466 9767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:10.656538 9767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.656607 9767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:10.656635 9767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.669596 9767 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:34033
I20260924 20:17:10.669829 9926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:34033 every 8 connection(s)
I20260924 20:17:10.671218 9767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:10.680418 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9767
I20260924 20:17:10.680938 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:33595
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:36907
--webserver_port=41417
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:10.696796 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.674066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.129:57585)
0924 20:17:10.674919 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:10.674981 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:10.676089 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:10.678251 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:10.678261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:10.678634 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:10.679320 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:10.679341 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.681271 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.681275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:10.682725 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:10.682738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.684251 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.684256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:10.684465 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:10.685271 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:10.685296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:10.690198 (+ 4902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:10.693116 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:10.693123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:10.693141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:10.693550 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:10.694067 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:10.694071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:10.694073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:10.694222 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:10.695012 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:10.695019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:10.695400 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 20:17:10.695608 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 20:17:10.695977 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":121,"threads_started":1}
I20260924 20:17:10.698549 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.673912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:57585 (local address 127.29.83.190:36907)
0924 20:17:10.674588 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:10.674612 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:10.676445 (+ 1833us) server_negotiation.cc:407] Connection header received
0924 20:17:10.676778 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:10.676795 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:10.677259 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:10.677633 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:10.679528 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.681102 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.682894 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.683804 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:10.683989 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:10.690516 (+ 6527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:10.690524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:10.690544 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:10.690582 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:10.692950 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:10.693705 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:10.693712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:10.693718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:10.693940 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:10.694340 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:10.694344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:10.694346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:10.694812 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:10.694952 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:10.697476 (+ 2524us) server_negotiation.cc:299] Negotiation successful
0924 20:17:10.697717 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":125,"threads_started":1}
I20260924 20:17:10.700902 9927 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
I20260924 20:17:10.701221 9927 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:10.701912 9927 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:10.704156 9927 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36907 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:17:10.704303 9927 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:10.704511 9927 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:10.705436 9927 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:10.705664 9927 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:10.706590 9927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:17:10.834600 9931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:10.834960 9931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:10.835034 9931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:10.840051 9931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:10.840144 9931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:10.840236 9931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:10.847903 9931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:33595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=41417
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:10.849431 9931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:10.850878 9931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:10.862066 9939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.862069 9936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:10.862097 9937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:10.862828 9931 server_base.cc:1034] running on GCE node
I20260924 20:17:10.863299 9931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:10.863996 9931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:10.865319 9931 hybrid_clock.cc:648] HybridClock initialized: now 1790281030865303 us; error 38 us; skew 500 ppm
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:10.870280 9931 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:10.871603 9931 webserver.cc:460] Webserver started at http://127.29.83.130:41417/ using document root <none> and password file <none>
I20260924 20:17:10.872244 9931 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:10.872328 9931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:17:10 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:10.890662 9931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:10.906827 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.915267 9931 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.000s sys 0.002s
I20260924 20:17:10.915458 9931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "d8d40a71cf2c4b93b411abb081a26711"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "47f0fbe89a4b8331abeab47f435a7cdd"
server_key_iv: "014a2834cff594c7e8a4375b35ef78f0"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:10.916290 9931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:10.945452 9931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:10.950276 9931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:10.950567 9931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:10.951359 9931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:10.952517 9931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:10.952713 9931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.952844 9931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:10.952919 9931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:10.967940 9931 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:33595
I20260924 20:17:10.969803 9931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:10.973246 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 9931
I20260924 20:17:10.973817 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.83.148:48923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:42833
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:36907
--webserver_port=39449
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:10.976418 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:33595 every 8 connection(s)
Sep 24 20:17:11 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281030, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:11.013118 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.980366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:51867 (local address 127.29.83.190:36907)
0924 20:17:10.980519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:10.980526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:10.981022 (+ 496us) server_negotiation.cc:407] Connection header received
0924 20:17:10.982727 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:10.982734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:10.982857 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:10.982949 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:10.985017 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:10.986073 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.003988 (+ 17915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.004384 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.004497 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:11.008070 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:11.008075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:11.008079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:11.008118 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:11.010148 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.010913 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.010917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.010919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.011026 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.011515 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.011519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.011522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.011801 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:11.011909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:11.012758 (+ 849us) server_negotiation.cc:299] Negotiation successful
0924 20:17:11.012922 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:17:11.015403 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:10.979102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.130:51867)
0924 20:17:10.980832 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:10.980899 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:10.982372 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:10.983504 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:10.983519 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:10.983961 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:10.984799 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:10.984824 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.001922 (+ 17098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.001928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:11.003714 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:11.003739 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.004780 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.004785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:11.005072 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:11.005919 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:11.005945 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:11.007775 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:11.010316 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:11.010323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:11.010344 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:11.010756 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:11.011157 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:11.011161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:11.011163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:11.011386 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:11.012054 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:11.012061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:11.012440 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 20:17:11.013719 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0924 20:17:11.014150 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":178,"threads_started":1}
I20260924 20:17:11.017428 10061 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
I20260924 20:17:11.017709 10061 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.018517 10061 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:11.020464 10061 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36907 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:17:11.020588 10061 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.020808 10061 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:11.021816 10061 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.022014 10061 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:11.023056 10061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:17:11.152060 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:11.152496 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:11.152585 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:11.156874 10063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:11.156982 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:11.157081 10063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:11.164352 10063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.83.148:48923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:42833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=39449
--tserver_master_addrs=127.29.83.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:11.165635 10063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:11.167160 10063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:11.178529 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:11.178514 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:11.178521 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:11.178786 10063 server_base.cc:1034] running on GCE node
I20260924 20:17:11.179471 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:11.180177 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:11.181360 10063 hybrid_clock.cc:648] HybridClock initialized: now 1790281031181324 us; error 48 us; skew 500 ppm
Sep 24 20:17:11 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:11.185920 10063 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:11.187228 10063 webserver.cc:460] Webserver started at http://127.29.83.131:39449/ using document root <none> and password file <none>
I20260924 20:17:11.187870 10063 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:11.187987 10063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:17:11 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:11.209671 10063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:17:11.213210 10081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:11.215088 10063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:17:11.215247 10063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "8e5ffe38913d43ba862828b725f6928f"
format_stamp: "Formatted at 2026-09-24 20:17:09 on dist-test-slave-52fr"
server_key: "9c981145a32a4415ae907d8634ac579b"
server_key_iv: "c413d2676f84b8646e622c40dfdbae9c"
server_key_version: "kuduclusterkey@0"
I20260924 20:17:11.216058 10063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:11.235082 10063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:11.239244 10063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:11.239483 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:11.240159 10063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:11.241068 10063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:11.241138 10063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:11.241209 10063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:11.241242 10063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:11.252352 10063 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42833
I20260924 20:17:11.252410 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42833 every 8 connection(s)
I20260924 20:17:11.253794 10063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:11.259595 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10063
Sep 24 20:17:11 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:11.272338 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:11.257839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:38691 (local address 127.29.83.190:36907)
0924 20:17:11.257985 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:11.257990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:11.258262 (+ 272us) server_negotiation.cc:407] Connection header received
0924 20:17:11.259430 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:11.259435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:11.259501 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:11.259564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:11.261239 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.262088 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.263931 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.264298 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.264398 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:11.267864 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:11.267870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:11.267875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:11.267912 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:11.269802 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.270471 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.270475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.270477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.270566 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.271003 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.271007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.271009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.271244 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:11.271360 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:11.272031 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 20:17:11.272168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:17:11.274151 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:11.256627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36907 (local address 127.29.83.131:38691)
0924 20:17:11.258102 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:11.258162 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:11.259136 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:11.260010 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:11.260019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:11.260391 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:11.261063 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:11.261082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.262259 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.262263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:11.263782 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:11.263793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.264618 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.264622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:11.264822 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:11.265519 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:11.265541 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:11.267331 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:11.269923 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:11.269930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:11.269952 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:11.270331 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:11.270679 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:11.270682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:11.270685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:11.270883 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:11.271380 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:11.271387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:11.271734 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 20:17:11.272618 (+ 884us) client_negotiation.cc:240] Negotiation successful
0924 20:17:11.272992 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":152,"threads_started":1}
I20260924 20:17:11.275548 10195 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36907
I20260924 20:17:11.275808 10195 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.276405 10195 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:11.278255 10195 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36907 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:17:11.278400 10195 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.278579 10195 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:11.279500 10195 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.279680 10195 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
W20260924 20:17:11.280493 10195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:17:11.285586 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:11.267488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.29.83.190:36907)
0924 20:17:11.267827 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:11.267836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:11.268847 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 20:17:11.268893 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:11.268896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:11.268947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:11.269068 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:11.271404 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.272227 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.280137 (+ 7910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:11.280489 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:11.280595 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:11.282244 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:11.282248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:11.282251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:11.282277 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:11.283928 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.284352 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.284355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.284358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.284447 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:11.284718 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:11.284721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:11.284723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:11.284962 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:11.285178 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:11.285266 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 20:17:11.285397 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":118,"threads_started":1}
Sep 24 20:17:11 dist-test-slave-52fr krb5kdc[8733](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:11.706893 9927 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:11.707315 9927 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:11.708648 9729 ts_manager.cc:194] Registered new tserver with Master: 0502eef23347409092aff514f2d06fe7 (127.29.83.129:34033)
I20260924 20:17:11.710129 9729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:57585
I20260924 20:17:12.023283 10061 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:12.023613 10061 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:12.024490 9729 ts_manager.cc:194] Registered new tserver with Master: d8d40a71cf2c4b93b411abb081a26711 (127.29.83.130:33595)
I20260924 20:17:12.025120 9729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:51867
Sep 24 20:17:12 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:12.280761 10195 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:12.281098 10195 heartbeater.cc:511] Master 127.29.83.190:36907 requested a full tablet report, sending...
I20260924 20:17:12.282095 9729 ts_manager.cc:194] Registered new tserver with Master: 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833)
I20260924 20:17:12.282923 9729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:38691
I20260924 20:17:12.289290 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:17:12.302485 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.291241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.29.83.190:36907)
0924 20:17:12.291559 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.291565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.291927 (+ 362us) server_negotiation.cc:407] Connection header received
0924 20:17:12.292060 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.292063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.292137 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.292234 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:12.293171 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.294123 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.295466 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.295867 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.295958 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.298037 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:12.298041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:12.298044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:12.298077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:12.300258 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:12.300862 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:12.300866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:12.300868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:12.300971 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:12.301327 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:12.301331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:12.301333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:12.301577 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:12.301692 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.302148 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.302297 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260924 20:17:12.307685 9729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:17:12.313279 9729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:17:12.344300 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.328099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.29.83.129:34033)
0924 20:17:12.330885 (+ 2786us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.330893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.330919 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:17:12.331001 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.331006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.331245 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.331423 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:12.331991 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.335839 (+ 3848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.340721 (+ 4882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.341672 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.343725 (+ 2053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.343927 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.343988 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.344101 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2672,"thread_start_us":83,"threads_started":1}
I20260924 20:17:12.345027 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.327690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34033 (local address 127.0.0.1:33920)
0924 20:17:12.330509 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.330523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.330612 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.331643 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:12.331650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:12.331677 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:12.331835 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.331843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.335979 (+ 4136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.335982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.340289 (+ 4307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.340305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.343194 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.343198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.343377 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.343451 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 20:17:12.343562 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:17:12.344817 (+ 1255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2755,"mutex_wait_us":1064,"thread_start_us":96,"threads_started":1}
I20260924 20:17:12.347021 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.328874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42833 (local address 127.0.0.1:34002)
0924 20:17:12.334536 (+ 5662us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.334552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.334649 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.335459 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:12.335462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:12.340472 (+ 5010us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:12.340574 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.340581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.342336 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.342340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.343749 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.343763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.346495 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.346499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.346676 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.346704 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:17:12.346786 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:17:12.346865 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5582,"mutex_wait_us":50,"thread_start_us":82,"threads_started":1}
I20260924 20:17:12.347529 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.329253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.29.83.131:42833)
0924 20:17:12.330360 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.330368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.334808 (+ 4440us) server_negotiation.cc:407] Connection header received
0924 20:17:12.334942 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.334949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.335156 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.335299 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:12.340706 (+ 5407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.342171 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.343921 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.346109 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.346246 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.346423 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.347253 (+ 830us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.347379 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":94,"threads_started":1}
Sep 24 20:17:12 dist-test-slave-52fr krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281031, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for HTTP/127.29.83.148@KRBTEST.COM
I20260924 20:17:12.351263 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.326046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:33595 (local address 127.0.0.1:52990)
0924 20:17:12.329847 (+ 3801us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.329909 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.330046 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.333728 (+ 3682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:12.333731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:12.333751 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:12.333920 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.333928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.345645 (+ 11717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.345648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.346987 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.346998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.350802 (+ 3804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.350808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.350925 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.350955 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:17:12.351032 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:17:12.351112 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3579,"mutex_wait_us":60,"thread_start_us":113,"threads_started":1}
I20260924 20:17:12.351563 10129 tablet_service.cc:1467] Processing CreateTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f (DEFAULT_TABLE table=test-table [id=f3eaf688fefd4554b046acc11199f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:12.352344 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.326896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.29.83.130:33595)
0924 20:17:12.332984 (+ 6088us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.332991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.333019 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:17:12.333097 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.333103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.333381 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.333543 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:12.341903 (+ 8360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.343040 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.347505 (+ 4465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.349577 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.351761 (+ 2184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.351952 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.352015 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.352195 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5967,"thread_start_us":94,"threads_started":1}
I20260924 20:17:12.352850 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406c4822664d4e71b7df5185ad8f5f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:12.352865 9861 tablet_service.cc:1467] Processing CreateTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f (DEFAULT_TABLE table=test-table [id=f3eaf688fefd4554b046acc11199f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:12.354065 9861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406c4822664d4e71b7df5185ad8f5f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:12.355769 9985 tablet_service.cc:1467] Processing CreateTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f (DEFAULT_TABLE table=test-table [id=f3eaf688fefd4554b046acc11199f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:12.356873 9985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406c4822664d4e71b7df5185ad8f5f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:12.367830 10230 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Bootstrap starting.
I20260924 20:17:12.370555 10230 tablet_bootstrap.cc:654] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:12.371587 10230 log.cc:824] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:12.374356 10231 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Bootstrap starting.
I20260924 20:17:12.374548 10230 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: No bootstrap required, opened a new log
I20260924 20:17:12.374768 10230 ts_tablet_manager.cc:1397] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:17:12.376889 10231 tablet_bootstrap.cc:654] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:12.377748 10231 log.cc:824] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:12.379128 10230 raft_consensus.cc:348] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.379470 10230 raft_consensus.cc:374] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:12.379534 10230 raft_consensus.cc:729] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d40a71cf2c4b93b411abb081a26711, State: Initialized, Role: FOLLOWER
I20260924 20:17:12.380421 10230 consensus_queue.cc:260] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.382150 10230 ts_tablet_manager.cc:1428] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:17:12.382200 10229 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Bootstrap starting.
I20260924 20:17:12.382695 10061 heartbeater.cc:503] Master 127.29.83.190:36907 was elected leader, sending a full tablet report...
I20260924 20:17:12.384673 10229 tablet_bootstrap.cc:654] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:12.385550 10229 log.cc:824] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:12.387362 10234 raft_consensus.cc:482] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:12.387518 10234 raft_consensus.cc:504] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.389876 10234 leader_election.cc:288] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0502eef23347409092aff514f2d06fe7 (127.29.83.129:34033), 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833)
I20260924 20:17:12.395016 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.389685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:55875 (local address 127.29.83.131:42833)
0924 20:17:12.389839 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.389844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.390578 (+ 734us) server_negotiation.cc:407] Connection header received
0924 20:17:12.390834 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.390838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.390891 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.390944 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:12.391429 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.392161 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.393590 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.394211 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.394315 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.394385 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.394785 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.394879 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:17:12.395758 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.389541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42833 (local address 127.29.83.130:55875)
0924 20:17:12.390432 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.390446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.390703 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.391098 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:12.391101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:12.391125 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:12.391264 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.391272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.392306 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.392309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.393451 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.393462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.394519 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.394523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.394626 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.394652 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:17:12.395491 (+ 839us) client_negotiation.cc:240] Negotiation successful
0924 20:17:12.395592 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":88,"threads_started":1}
I20260924 20:17:12.396459 10231 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: No bootstrap required, opened a new log
I20260924 20:17:12.396695 10231 ts_tablet_manager.cc:1397] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20260924 20:17:12.396881 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406c4822664d4e71b7df5185ad8f5f0f" candidate_uuid: "d8d40a71cf2c4b93b411abb081a26711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ffe38913d43ba862828b725f6928f" is_pre_election: true
W20260924 20:17:12.399482 9951 leader_election.cc:341] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833): Illegal state: must be running to vote when last-logged opid is not known
I20260924 20:17:12.400935 10231 raft_consensus.cc:348] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.401248 10231 raft_consensus.cc:374] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:12.401315 10231 raft_consensus.cc:729] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0502eef23347409092aff514f2d06fe7, State: Initialized, Role: FOLLOWER
I20260924 20:17:12.403367 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.389147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:42131 (local address 127.29.83.129:34033)
0924 20:17:12.389289 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.389296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.390130 (+ 834us) server_negotiation.cc:407] Connection header received
0924 20:17:12.390248 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.390251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.390323 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.390395 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:12.390933 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.391749 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.401896 (+ 10147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.402588 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.402729 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.402784 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.403142 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.403230 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:17:12.404206 10229 tablet_bootstrap.cc:492] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: No bootstrap required, opened a new log
I20260924 20:17:12.404423 10229 ts_tablet_manager.cc:1397] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260924 20:17:12.407519 10229 raft_consensus.cc:348] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.407799 10229 raft_consensus.cc:374] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:12.407848 10229 raft_consensus.cc:729] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ffe38913d43ba862828b725f6928f, State: Initialized, Role: FOLLOWER
I20260924 20:17:12.408181 10231 consensus_queue.cc:260] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.409375 10231 ts_tablet_manager.cc:1428] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260924 20:17:12.409798 9927 heartbeater.cc:503] Master 127.29.83.190:36907 was elected leader, sending a full tablet report...
I20260924 20:17:12.412510 10229 consensus_queue.cc:260] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.412993 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.388992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34033 (local address 127.29.83.130:42131)
0924 20:17:12.389947 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.390025 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.390148 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.390566 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:12.390569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:12.390589 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:12.390780 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.390788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.391905 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.391908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.393221 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:12.393232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.402861 (+ 9629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.402866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:12.402987 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.403015 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:17:12.412554 (+ 9539us) client_negotiation.cc:240] Negotiation successful
0924 20:17:12.412798 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":3,"spinlock_wait_cycles":21376,"thread_start_us":119,"threads_started":1}
I20260924 20:17:12.413655 10229 ts_tablet_manager.cc:1428] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260924 20:17:12.413863 9881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406c4822664d4e71b7df5185ad8f5f0f" candidate_uuid: "d8d40a71cf2c4b93b411abb081a26711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0502eef23347409092aff514f2d06fe7" is_pre_election: true
I20260924 20:17:12.414147 9881 raft_consensus.cc:2454] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d40a71cf2c4b93b411abb081a26711 in term 0.
I20260924 20:17:12.414535 10195 heartbeater.cc:503] Master 127.29.83.190:36907 was elected leader, sending a full tablet report...
I20260924 20:17:12.416621 9951 leader_election.cc:302] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0502eef23347409092aff514f2d06fe7, d8d40a71cf2c4b93b411abb081a26711; no voters: 8e5ffe38913d43ba862828b725f6928f
I20260924 20:17:12.417191 10234 raft_consensus.cc:2781] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:12.417285 10234 raft_consensus.cc:482] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:12.417335 10234 raft_consensus.cc:3037] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:12.418828 10234 raft_consensus.cc:504] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
I20260924 20:17:12.419210 10234 leader_election.cc:288] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [CANDIDATE]: Term 1 election: Requested vote from peers 0502eef23347409092aff514f2d06fe7 (127.29.83.129:34033), 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833)
I20260924 20:17:12.420828 9881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406c4822664d4e71b7df5185ad8f5f0f" candidate_uuid: "d8d40a71cf2c4b93b411abb081a26711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0502eef23347409092aff514f2d06fe7"
I20260924 20:17:12.420960 9881 raft_consensus.cc:3037] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:12.422394 9881 raft_consensus.cc:2454] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d40a71cf2c4b93b411abb081a26711 in term 1.
I20260924 20:17:12.422994 9951 leader_election.cc:302] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0502eef23347409092aff514f2d06fe7, d8d40a71cf2c4b93b411abb081a26711; no voters:
I20260924 20:17:12.423239 10234 raft_consensus.cc:2781] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:12.423480 10234 raft_consensus.cc:686] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [term 1 LEADER]: Becoming Leader. State: Replica: d8d40a71cf2c4b93b411abb081a26711, State: Running, Role: LEADER
I20260924 20:17:12.423839 10234 consensus_queue.cc:237] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } }
W20260924 20:17:12.424685 9928 tablet.cc:2369] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:17:12.425360 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406c4822664d4e71b7df5185ad8f5f0f" candidate_uuid: "d8d40a71cf2c4b93b411abb081a26711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ffe38913d43ba862828b725f6928f"
I20260924 20:17:12.425529 10149 raft_consensus.cc:3037] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:12.427031 10149 raft_consensus.cc:2454] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d40a71cf2c4b93b411abb081a26711 in term 1.
I20260924 20:17:12.432317 9729 catalog_manager.cc:5495] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d40a71cf2c4b93b411abb081a26711 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "d8d40a71cf2c4b93b411abb081a26711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d40a71cf2c4b93b411abb081a26711" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 33595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260924 20:17:12.471897 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.466474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.29.83.130:33595)
0924 20:17:12.466597 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:12.466602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:12.466618 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:17:12.466714 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:12.466717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:12.466793 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:12.466933 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:17:12.467368 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.468181 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.469497 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:12.469885 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:12.469987 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:12.470437 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:17:12.471048 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:17:12.471167 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:12.471652 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 20:17:12.471717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:17:12.479173 10149 raft_consensus.cc:1261] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 1 FOLLOWER]: Refusing update from remote peer d8d40a71cf2c4b93b411abb081a26711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 20:17:12.479960 10062 tablet.cc:2369] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:17:12.479978 10234 consensus_queue.cc:1035] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ffe38913d43ba862828b725f6928f" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:12.481377 9881 raft_consensus.cc:1261] T 406c4822664d4e71b7df5185ad8f5f0f P 0502eef23347409092aff514f2d06fe7 [term 1 FOLLOWER]: Refusing update from remote peer d8d40a71cf2c4b93b411abb081a26711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:12.482179 10241 consensus_queue.cc:1035] T 406c4822664d4e71b7df5185ad8f5f0f P d8d40a71cf2c4b93b411abb081a26711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0502eef23347409092aff514f2d06fe7" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:12.499260 10244 mvcc.cc:205] Tried to move back new op lower bound from 7332991109024821248 to 7332991108823535616. Current Snapshot: MvccSnapshot[applied={T|T < 7332991109024821248}]
I20260924 20:17:12.500926 10245 mvcc.cc:205] Tried to move back new op lower bound from 7332991109024821248 to 7332991108823535616. Current Snapshot: MvccSnapshot[applied={T|T < 7332991109024821248}]
I20260924 20:17:12.513350 9725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:12.513549 9725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:12.513952 10246 mvcc.cc:205] Tried to move back new op lower bound from 7332991109024821248 to 7332991108823535616. Current Snapshot: MvccSnapshot[applied={T|T < 7332991109024821248}]
I20260924 20:17:12.516739 9725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a121b25e39f442b9029d36c74b44cf9: Sending DeleteTablet for 3 replicas of tablet 406c4822664d4e71b7df5185ad8f5f0f
I20260924 20:17:12.517496 9985 tablet_service.cc:1514] Processing DeleteTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:52990
I20260924 20:17:12.518424 9861 tablet_service.cc:1514] Processing DeleteTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:33920
I20260924 20:17:12.519279 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet 406c4822664d4e71b7df5185ad8f5f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:12 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:34002
I20260924 20:17:12.520399 10255 tablet_replica.cc:330] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: stopping tablet replica
I20260924 20:17:12.520704 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9767
I20260924 20:17:12.521549 10255 raft_consensus.cc:2229] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:12.521935 10255 raft_consensus.cc:2258] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:12.523561 10255 ts_tablet_manager.cc:1905] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:12.527187 10255 ts_tablet_manager.cc:1918] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:12.527311 10255 log.cc:1196] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/406c4822664d4e71b7df5185ad8f5f0f
I20260924 20:17:12.527648 10255 ts_tablet_manager.cc:1939] T 406c4822664d4e71b7df5185ad8f5f0f P 8e5ffe38913d43ba862828b725f6928f: Deleting consensus metadata
W20260924 20:17:12.529145 9716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:34033 (error 108)
I20260924 20:17:12.529417 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9931
I20260924 20:17:12.529666 9716 catalog_manager.cc:4841] TS 8e5ffe38913d43ba862828b725f6928f (127.29.83.131:42833): tablet 406c4822664d4e71b7df5185ad8f5f0f (table test-table [id=f3eaf688fefd4554b046acc11199f6be]) successfully deleted
I20260924 20:17:12.531080 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:12.530793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:34033 (local address 127.0.0.1:33930)
0924 20:17:12.530911 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.530985 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 20:17:12.531252 9716 catalog_manager.cc:4584] TS 0502eef23347409092aff514f2d06fe7 (127.29.83.129:34033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 406c4822664d4e71b7df5185ad8f5f0f: Network error: Client connection negotiation failed: client connection to 127.29.83.129:34033: connect: Connection refused (error 111)
W20260924 20:17:12.538232 10227 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:17:12.537524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:33595 (local address 127.0.0.1:53008)
0924 20:17:12.537719 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:12.537735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:12.537873 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:12.538110 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.130:33595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 20:17:12.538851 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10063
I20260924 20:17:12.546334 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 9696
I20260924 20:17:12.555308 30030 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 20:17:12.901286 30030 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 20:17:12.901413 30030 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:17:13.246992 30030 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:17:13.247 UTC [8763] LOG: received smart shutdown request
2026-09-24 20:17:13.250 UTC [8763] LOG: background worker "logical replication launcher" (PID 8772) exited with exit code 1
2026-09-24 20:17:13.250 UTC [8767] LOG: shutting down
2026-09-24 20:17:13.442 UTC [8763] LOG: database system is shut down
2026-09-24T20:17:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49675 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:17:13 dist-test-slave-52fr krb5kdc[10265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:17:13 dist-test-slave-52fr krb5kdc[10265](info): set up 2 sockets
Sep 24 20:17:13 dist-test-slave-52fr krb5kdc[10265](info): commencing operation
krb5kdc: starting...
W20260924 20:17:14.610069 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:17:14 dist-test-slave-52fr krb5kdc[10265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:17:14.736753 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36895
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:14.850350 10278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:14.850705 10278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:14.850800 10278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:14.854996 10278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:14.855093 10278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:14.855131 10278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:14.855196 10278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:14.855234 10278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:14.860688 10278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:14.861969 10278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:14.863021 10278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:14.871750 10284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:14.871748 10283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:14.871825 10278 server_base.cc:1034] running on GCE node
W20260924 20:17:14.871748 10286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:14.872646 10278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:14.873824 10278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:14.875008 10278 hybrid_clock.cc:648] HybridClock initialized: now 1790281034874975 us; error 47 us; skew 500 ppm
Sep 24 20:17:14 dist-test-slave-52fr krb5kdc[10265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:14.878055 10278 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:14.879024 10278 webserver.cc:460] Webserver started at http://127.29.83.190:33479/ using document root <none> and password file <none>
I20260924 20:17:14.879607 10278 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:14.879673 10278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:14.879894 10278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:14.882047 10278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "9477d987e8a042e9ad1e6efcd7a94662"
format_stamp: "Formatted at 2026-09-24 20:17:14 on dist-test-slave-52fr"
server_key: "294ab9a12707793b4b606ec7e4dcd551"
server_key_iv: "82e6b2c6a1e10d8e728072abc73a821e"
server_key_version: "encryptionkey@0"
I20260924 20:17:14.882680 10278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "9477d987e8a042e9ad1e6efcd7a94662"
format_stamp: "Formatted at 2026-09-24 20:17:14 on dist-test-slave-52fr"
server_key: "294ab9a12707793b4b606ec7e4dcd551"
server_key_iv: "82e6b2c6a1e10d8e728072abc73a821e"
server_key_version: "encryptionkey@0"
I20260924 20:17:14.886693 10278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:17:14.889253 10293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:14.890581 10278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:17:14.890728 10278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "9477d987e8a042e9ad1e6efcd7a94662"
format_stamp: "Formatted at 2026-09-24 20:17:14 on dist-test-slave-52fr"
server_key: "294ab9a12707793b4b606ec7e4dcd551"
server_key_iv: "82e6b2c6a1e10d8e728072abc73a821e"
server_key_version: "encryptionkey@0"
I20260924 20:17:14.890853 10278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:14.911906 10278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:14.915740 10278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:14.915998 10278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:14.923923 10278 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36895
I20260924 20:17:14.924026 10345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36895 every 8 connection(s)
I20260924 20:17:14.925027 10278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:14.928025 10346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:14.931986 10346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:14.933346 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10278
I20260924 20:17:14.933532 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:17:14.933920 30030 external_mini_cluster.cc:1428] Setting key 0360938b0d2d5311614a44edcef6ff7b
WARNING: no policy specified for kudu/127.29.83.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.189@KRBTEST.COM" created.
Sep 24 20:17:14 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:14.951242 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:14.934592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36895 (local address 127.0.0.1:48290)
0924 20:17:14.935222 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:14.935295 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:14.936204 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:14.936917 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:14.936922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:14.937382 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:14.937701 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:14.937716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:14.938907 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:14.938910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:14.939636 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:14.939646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:14.940275 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:14.940279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:14.940393 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:14.941100 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:14.941118 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:14.945185 (+ 4067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:14.947853 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:14.947863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:14.947873 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:14.948119 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:14.948571 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:14.948574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:14.948575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:14.948740 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:14.949604 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:14.949610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:14.949862 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 20:17:14.950032 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 20:17:14.950335 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":160,"threads_started":1}
I20260924 20:17:14.951247 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:14.934592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.29.83.190:36895)
0924 20:17:14.935607 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:14.935618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:14.935667 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:17:14.936513 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:14.936519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:14.936732 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:14.936921 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:14.937897 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:14.938778 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:14.939751 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:14.940121 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:14.940290 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:14.945406 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:14.945413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:14.945426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:14.945452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:14.947709 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:14.948242 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:14.948252 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:14.948266 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:14.948445 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:14.948847 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:14.948851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:14.948853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:14.949470 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:14.949629 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:14.950151 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 20:17:14.950448 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":890,"spinlock_wait_cycles":43776,"thread_start_us":160,"threads_started":1}
I20260924 20:17:14.954015 10346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:14.954802 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:14.954547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60498)
0924 20:17:14.954666 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:14.954743 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:17:14.955055 10295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
I20260924 20:17:14.955904 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:14.955659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60510)
0924 20:17:14.955796 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:14.955852 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 20:17:14.956039 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:35007: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
I20260924 20:17:14.962167 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:35007
--webserver_interface=127.29.83.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:15.005347 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } attempt: 1
I20260924 20:17:15.006533 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.006225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60512)
0924 20:17:15.006381 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.006453 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:17:15.007296 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.007064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60524)
0924 20:17:15.007174 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.007229 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:17:15.007432 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:35007: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
I20260924 20:17:15.075626 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } attempt: 2
I20260924 20:17:15.076604 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.076273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60538)
0924 20:17:15.076441 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.076518 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 20:17:15.077461 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.077171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60548)
0924 20:17:15.077288 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.077369 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 20:17:15.077589 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:35007: Network error: Client connection negotiation failed: client connection to 127.29.83.189:35007: connect: Connection refused (error 111)
W20260924 20:17:15.078243 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:15.078555 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:15.078650 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:15.083030 10353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:15.083124 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:15.083164 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:15.083215 10353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:15.083251 10353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:15.089125 10353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.83.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:15.090418 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:15.091535 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:15.100721 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:15.100749 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:15.100754 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:15.101115 10353 server_base.cc:1034] running on GCE node
I20260924 20:17:15.101871 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:15.103055 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:15.104288 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790281035104262 us; error 38 us; skew 500 ppm
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:15.107502 10353 init.cc:377] Logged in from keytab as kudu/127.29.83.189@KRBTEST.COM (short username kudu)
I20260924 20:17:15.108526 10353 webserver.cc:460] Webserver started at http://127.29.83.189:35347/ using document root <none> and password file <none>
I20260924 20:17:15.109110 10353 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:15.109205 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:15.109441 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:15.111536 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data/instance:
uuid: "ba221a077346418ea1d2b48d5c464bd1"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "a24e9ed75278ea9ef94f5eaf1691fa7e"
server_key_iv: "fbcfda7c79d8ef1bf434598062aea501"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.112097 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal/instance:
uuid: "ba221a077346418ea1d2b48d5c464bd1"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "a24e9ed75278ea9ef94f5eaf1691fa7e"
server_key_iv: "fbcfda7c79d8ef1bf434598062aea501"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.115875 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:17:15.118393 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:15.119737 10353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:17:15.119838 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "ba221a077346418ea1d2b48d5c464bd1"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "a24e9ed75278ea9ef94f5eaf1691fa7e"
server_key_iv: "fbcfda7c79d8ef1bf434598062aea501"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.119926 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:15.137094 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:15.140503 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:15.140668 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:15.148001 10353 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.189:35007
I20260924 20:17:15.148021 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.189:35007 every 8 connection(s)
I20260924 20:17:15.149104 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
I20260924 20:17:15.152069 10421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:15.156018 10421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:15.158937 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10353
I20260924 20:17:15.159096 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-1/wal/instance
I20260924 20:17:15.159415 30030 external_mini_cluster.cc:1428] Setting key 8864b4fd7852c0b4d36574853cbbd054
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.188@KRBTEST.COM; defaulting to no policy
I20260924 20:17:15.168766 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.188@KRBTEST.COM" created.
I20260924 20:17:15.171568 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.158145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.29.83.190:36895)
0924 20:17:15.158319 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.158324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.159190 (+ 866us) server_negotiation.cc:407] Connection header received
0924 20:17:15.160270 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.160281 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.160348 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.160452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.161663 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.162344 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.163529 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.163787 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.163906 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.166812 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.166816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.166819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.166858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.168597 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.169367 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.169371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.169373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.169469 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.169947 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.169953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.169958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.170253 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.170363 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.171138 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.171313 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:17:15.172793 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.158528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36895 (local address 127.0.0.1:48300)
0924 20:17:15.159032 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.159085 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.160050 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.160630 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.160637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.161004 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.161479 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.161492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.162481 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.162487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.163385 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.163395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.163962 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.163966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.164148 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.164792 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.164810 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.166600 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.168803 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.168811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.168828 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.169211 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.169585 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.169588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.169590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.169840 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.170382 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.170393 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.170712 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.170967 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.171331 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":149,"threads_started":1}
I20260924 20:17:15.174670 10421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:17:15.181068 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.169241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60556)
0924 20:17:15.169380 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.169399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.169540 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.170433 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.170437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.170618 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.170768 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.170778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.172156 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.172160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.172996 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.173007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.173828 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.173832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.173950 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.174507 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.174522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.176048 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.179166 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.179169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.179172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.179444 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.179833 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.179836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.179837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.179921 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.180587 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.180590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.180694 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.180797 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.180935 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 20:17:15.181535 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.169364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.29.83.189:35007)
0924 20:17:15.169918 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.169932 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.169958 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:17:15.170012 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.170017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.170256 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.170459 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.170900 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.172017 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.173149 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.173673 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.173848 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.176194 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.176203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.176218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.176247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.179016 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.179564 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.179569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.179570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.179709 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.180017 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.180021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.180023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.180420 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.180554 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.180876 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.180994 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":169,"threads_started":1}
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:17:15.183578 10346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:15.184314 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.184062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:55110)
0924 20:17:15.184184 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.184241 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 20:17:15.184975 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.184718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:55112)
0924 20:17:15.184884 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.184929 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 20:17:15.185191 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.187458 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.175240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:60570)
0924 20:17:15.175360 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.175374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.175458 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.176502 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.176508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.176688 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.176817 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.176823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.177582 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.177588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.178449 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.178461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.179129 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.179136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.179248 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.179729 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.179748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.183320 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.185898 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.185901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.185903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.186101 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.186389 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.186392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.186393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.186472 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.186939 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.186941 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.187032 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.187151 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.187302 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 20:17:15.187546 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.175854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.29.83.189:35007)
0924 20:17:15.176228 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.176239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.176255 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:17:15.176296 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.176300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.176358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.176463 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.176931 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.177478 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.178583 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.178999 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.179106 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.183574 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.183577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.183579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.183599 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.185717 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.186207 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.186213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.186215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.186293 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.186554 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.186559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.186561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.186825 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.186939 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.187197 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.187386 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260924 20:17:15.188341 10421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
I20260924 20:17:15.189116 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.188876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:55116)
0924 20:17:15.188987 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.189059 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:17:15.189337 10372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.189715 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:46103
--webserver_interface=127.29.83.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:15.190521 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.190273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:55120)
0924 20:17:15.190413 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.190462 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 20:17:15.190686 10421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.216387 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 1
I20260924 20:17:15.217352 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.216999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50570)
0924 20:17:15.217184 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.217250 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 20:17:15.218277 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.217985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50574)
0924 20:17:15.218145 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.218214 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 20:17:15.218444 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.239957 10421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 1
I20260924 20:17:15.240942 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.240615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50576)
0924 20:17:15.240788 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.240859 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 20:17:15.241741 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.241510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50582)
0924 20:17:15.241632 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.241687 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 20:17:15.241958 10421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.293632 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 2
I20260924 20:17:15.294565 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.294264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50590)
0924 20:17:15.294412 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.294485 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 20:17:15.295359 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.295145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50600)
0924 20:17:15.295247 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.295305 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:17:15.295488 10346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
W20260924 20:17:15.306324 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:15.306656 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:15.306768 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:17:15.310112 10421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 2
I20260924 20:17:15.310981 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.310678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50608)
0924 20:17:15.310830 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.310899 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:17:15.311273 10429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:15.311343 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:15.311378 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:15.311422 10429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:15.311455 10429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:15.311770 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.311554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50614)
0924 20:17:15.311662 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.311715 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:17:15.311945 10421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46103: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46103: connect: Connection refused (error 111)
I20260924 20:17:15.318711 10429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36895,127.29.83.189:35007,127.29.83.188:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.83.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:15.319939 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:15.321019 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:15.330614 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:15.330662 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:15.330832 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:15.331022 10429 server_base.cc:1034] running on GCE node
I20260924 20:17:15.331668 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:15.333042 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:15.334218 10429 hybrid_clock.cc:648] HybridClock initialized: now 1790281035334201 us; error 42 us; skew 500 ppm
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:15.338723 10429 init.cc:377] Logged in from keytab as kudu/127.29.83.188@KRBTEST.COM (short username kudu)
I20260924 20:17:15.339877 10429 webserver.cc:460] Webserver started at http://127.29.83.188:36357/ using document root <none> and password file <none>
I20260924 20:17:15.340616 10429 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:15.340684 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:15.340864 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:15.342948 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data/instance:
uuid: "9bcf8ba878a74dd09368cd4781d1728d"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "94cef4b7fde487a4b07cb3e4639cd31e"
server_key_iv: "aa80cf04dd9994b39a72442a356131ee"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.343453 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal/instance:
uuid: "9bcf8ba878a74dd09368cd4781d1728d"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "94cef4b7fde487a4b07cb3e4639cd31e"
server_key_iv: "aa80cf04dd9994b39a72442a356131ee"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.347295 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:17:15.349736 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:15.351054 10429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:17:15.351169 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "9bcf8ba878a74dd09368cd4781d1728d"
format_stamp: "Formatted at 2026-09-24 20:17:15 on dist-test-slave-52fr"
server_key: "94cef4b7fde487a4b07cb3e4639cd31e"
server_key_iv: "aa80cf04dd9994b39a72442a356131ee"
server_key_version: "encryptionkey@0"
I20260924 20:17:15.351266 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:15.366181 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:15.369079 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:15.369237 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:15.376938 10429 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.188:46103
I20260924 20:17:15.377051 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.188:46103 every 8 connection(s)
I20260924 20:17:15.378032 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
I20260924 20:17:15.380784 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:15.384687 10497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:15.386853 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10429
I20260924 20:17:15.386981 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280784619584-30030-0/minicluster-data/master-2/wal/instance
I20260924 20:17:15.387276 30030 external_mini_cluster.cc:1428] Setting key bee4de9dd7cead8e9a5699ce49b6f934
I20260924 20:17:15.394641 10346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 3
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:17:15.399953 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.386742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.29.83.190:36895)
0924 20:17:15.386891 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.386894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.388097 (+ 1203us) server_negotiation.cc:407] Connection header received
0924 20:17:15.389339 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.389343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.389404 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.389474 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.390470 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.390981 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.392194 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.392474 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.392553 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.395608 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.395612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.395615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.395643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.397533 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.398181 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.398184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.398186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.398273 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.398635 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.398638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.398640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.398898 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.398988 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.399692 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.399820 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:17:15.401129 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.388268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.29.83.188:46103)
0924 20:17:15.388577 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.388588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.388629 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:17:15.389089 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.389114 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.389395 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.389591 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.390403 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.391455 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.392302 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.392748 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.392860 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.396133 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.396143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.396163 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.396196 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.398232 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.398758 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.398764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.398765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.398854 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.399148 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.399151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.399153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.399476 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.399705 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.399887 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.400215 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"spinlock_wait_cycles":3072,"thread_start_us":123,"threads_started":1}
I20260924 20:17:15.401135 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.387269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36895 (local address 127.0.0.1:35056)
0924 20:17:15.387914 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.387976 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.389089 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.389562 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.389567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.390047 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.390298 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.390310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.391091 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.391095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.392072 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.392081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.392614 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.392618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.392790 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.393631 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.393648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.395456 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.397652 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.397663 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.397677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.398073 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.398384 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.398387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.398389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.398529 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.399042 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.399049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.399387 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.400119 (+ 732us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.400317 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"spinlock_wait_cycles":896,"thread_start_us":160,"threads_started":1}
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:17:15.402909 10497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:15.403162 10421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } attempt: 3
I20260924 20:17:15.407228 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.395123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50624)
0924 20:17:15.395327 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.395369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.395454 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.396512 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.396516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.396706 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.396830 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.396838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.397751 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.397754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.398631 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.398643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.399235 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.399239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.399321 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.400265 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.400279 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.401815 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.405587 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.405590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.405592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.405879 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.406209 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.406212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.406214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.406305 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.406730 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.406733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.406824 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.406903 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.407056 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 20:17:15.407966 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.395210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.29.83.188:46103)
0924 20:17:15.396283 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.396294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.396307 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:17:15.396344 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.396347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.396393 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.396490 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.396956 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.397562 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.398758 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.399097 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.399223 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.401956 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.401959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.401963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.401993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.405466 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.406000 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.406005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.406008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.406108 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.406411 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.406414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.406415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.406607 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.407506 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.407583 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.407759 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":989,"thread_start_us":157,"threads_started":1}
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:17:15.411850 10346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: Bootstrap starting.
I20260924 20:17:15.415285 10346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:15.415329 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.404092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.29.83.189:35007)
0924 20:17:15.404194 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.404197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.404344 (+ 147us) server_negotiation.cc:407] Connection header received
0924 20:17:15.404492 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.404497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.404546 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.404640 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.405134 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.405999 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407277 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.407649 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407721 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.411539 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.411547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.411552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.411582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.413193 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.413743 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.413747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.413749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.413870 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414222 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.414225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.414227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.414533 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.414601 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.414989 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.415123 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:17:15.415621 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.403845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.29.83.190:36895)
0924 20:17:15.404190 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.404194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.404208 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:17:15.404243 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.404245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.404285 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.404355 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.405214 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.405989 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407133 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.407444 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407533 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.410740 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.410744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.410747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.410772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.412505 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.413744 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.413747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.413749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.413890 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414221 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.414225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.414227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.414458 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.414653 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.415344 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.415476 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260924 20:17:15.415659 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.403592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50632)
0924 20:17:15.403859 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.403875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.403956 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.404333 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.404339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.404789 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.404894 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.404904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.406046 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.406049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.406899 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.406910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.407783 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.407877 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.409024 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.409040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.410949 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.414047 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.414052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.414297 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.414600 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.414605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.414689 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.415124 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.415141 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.415249 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.415344 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.415480 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260924 20:17:15.415836 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.403616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:35007 (local address 127.0.0.1:44450)
0924 20:17:15.404274 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.404287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.404397 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.404663 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.404668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.404892 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.404996 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.405003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.406375 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.406378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.407163 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.407172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.407873 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.407981 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.408364 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.408385 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.411395 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.413325 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.413329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.413331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.413615 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.414002 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.414007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.414111 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.414676 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.414679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.414780 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.415530 (+ 750us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.415680 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"spinlock_wait_cycles":7552}
I20260924 20:17:15.416349 10346 log.cc:824] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:15.416465 10497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:17:15.416666 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.403852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.29.83.188:46103)
0924 20:17:15.404040 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.404044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.404056 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:17:15.404094 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.404097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.404185 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.405013 (+ 828us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.405104 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.405926 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.407033 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.407653 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.409587 (+ 1934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.411116 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.411120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.411123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.411192 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.413920 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414422 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.414425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.414427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.414501 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.414795 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.414798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.414799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.415008 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.416269 (+ 1261us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.416351 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.416497 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":109}
I20260924 20:17:15.422420 10421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: Bootstrap starting.
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281035, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:17:15.424438 10346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: No bootstrap required, opened a new log
I20260924 20:17:15.425359 10421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:15.426488 10421 log.cc:824] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:17:15 dist-test-slave-52fr krb5kdc[10265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:17:15.428689 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.417238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46103 (local address 127.0.0.1:50642)
0924 20:17:15.417353 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:15.417368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:15.417442 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:15.417841 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:15.417846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:15.418015 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:15.418110 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.418117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.419055 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.419062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.419879 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:15.419892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.420661 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.420665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:15.420762 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.421339 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:15.421357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:15.422901 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:15.427112 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.427117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.427120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.427389 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.427675 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:15.427677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:15.427679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:15.427748 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:15.428153 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:15.428155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:15.428239 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 20:17:15.428369 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:17:15.428514 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 20:17:15.429191 10421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: No bootstrap required, opened a new log
I20260924 20:17:15.429231 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.417437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.29.83.188:46103)
0924 20:17:15.417524 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.417529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.417540 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:17:15.417571 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.417573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.417611 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.417698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.418224 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.418937 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.420012 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.420386 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.420496 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.423037 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.423041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.423044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.423067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.426973 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.427507 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.427510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.427512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.427590 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.427825 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.427828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.427829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.428030 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.428118 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.428975 (+ 857us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.429093 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 20:17:15.431803 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.417292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.29.83.189:35007)
0924 20:17:15.417389 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.417393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.417820 (+ 427us) server_negotiation.cc:407] Connection header received
0924 20:17:15.418012 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.418015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.418058 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.418106 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.419044 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.419849 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.421149 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.421516 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.421618 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.427776 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.427780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.427783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.427824 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.429483 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.430061 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.430065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.430066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.430155 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.430507 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.430510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.430512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.430726 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.430900 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.431534 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.431657 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 20:17:15.432382 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: Bootstrap starting.
I20260924 20:17:15.433461 10421 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.433398 10346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.433666 10346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:15.433665 10421 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:15.433732 10421 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba221a077346418ea1d2b48d5c464bd1, State: Initialized, Role: FOLLOWER
I20260924 20:17:15.433728 10346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9477d987e8a042e9ad1e6efcd7a94662, State: Initialized, Role: FOLLOWER
I20260924 20:17:15.434262 10346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.434262 10421 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.435467 10421 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:15.435575 10346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:15.437515 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:17:15.440527 10497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: Neither blocks nor log segments found. Creating new log.
W20260924 20:17:15.441107 10528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:17:15.441108 10523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:17:15.441184 10523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 20:17:15.441308 10528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:17:15.441460 10497 log.cc:824] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:15.442009 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.442127 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.442790 10506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.442924 10506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.445627 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: No bootstrap required, opened a new log
I20260924 20:17:15.444269 10353 master_runner.cc:186] Error getting master registration for 127.29.83.188:46103: OK, instance_id { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" instance_seqno: 1790281035374607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:17:15.447614 10353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:17:15.451179 10278 master_runner.cc:186] Error getting master registration for 127.29.83.188:46103: OK, instance_id { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" instance_seqno: 1790281035374607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:17:15.460109 10278 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:17:15.460570 10497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.460930 10497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:15.461014 10497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcf8ba878a74dd09368cd4781d1728d, State: Initialized, Role: FOLLOWER
I20260924 20:17:15.461660 10497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.462112 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.442642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.29.83.190:36895)
0924 20:17:15.442757 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.442761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.446023 (+ 3262us) server_negotiation.cc:407] Connection header received
0924 20:17:15.446210 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.446213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.446259 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.446310 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.453880 (+ 7570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.454620 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.455869 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.456194 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.456266 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.458501 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.458505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.458508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.458537 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.460146 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.460636 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.460640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.460642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.460726 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.461066 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.461071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.461074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.461294 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.461418 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.461748 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.461926 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:17:15.462864 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.449930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.29.83.189:35007)
0924 20:17:15.450152 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.450157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.450405 (+ 248us) server_negotiation.cc:407] Connection header received
0924 20:17:15.450603 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.450606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.450695 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.450749 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.451852 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.452639 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.453888 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.454227 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.454321 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.456395 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.456399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.456402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.456430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.458195 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.461314 (+ 3119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.461319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.461320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.461419 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.461735 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.461738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.461740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.461983 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.462094 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.462576 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.462701 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 20:17:15.463503 10497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:15.466392 10539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.466529 10539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.470656 10429 master_runner.cc:263] No leader master found from master 9bcf8ba878a74dd09368cd4781d1728d
I20260924 20:17:15.470722 10429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:17:15.472538 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:15.443181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.29.83.188:46103)
0924 20:17:15.443305 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:15.443309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:15.456945 (+ 13636us) server_negotiation.cc:407] Connection header received
0924 20:17:15.457125 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:15.457128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:15.457183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:15.457236 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:15.459243 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.460144 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.464331 (+ 4187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:15.464679 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:15.464751 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:15.469096 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:15.469100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:15.469103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:15.469129 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:15.470846 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.471308 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.471312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.471314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.471389 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:15.471663 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:15.471666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:15.471668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:15.471872 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:15.471974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:15.472288 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 20:17:15.472396 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:17:15.476889 10550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:17:15.476981 10550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:17:15.524086 10539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:15.524485 10539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.525700 10539 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9477d987e8a042e9ad1e6efcd7a94662 (127.29.83.190:36895), ba221a077346418ea1d2b48d5c464bd1 (127.29.83.189:35007)
I20260924 20:17:15.526835 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bcf8ba878a74dd09368cd4781d1728d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9477d987e8a042e9ad1e6efcd7a94662" is_pre_election: true
I20260924 20:17:15.527187 10320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bcf8ba878a74dd09368cd4781d1728d in term 0.
I20260924 20:17:15.527807 10448 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9477d987e8a042e9ad1e6efcd7a94662, 9bcf8ba878a74dd09368cd4781d1728d; no voters:
I20260924 20:17:15.528614 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bcf8ba878a74dd09368cd4781d1728d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba221a077346418ea1d2b48d5c464bd1" is_pre_election: true
I20260924 20:17:15.528949 10395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bcf8ba878a74dd09368cd4781d1728d in term 0.
I20260924 20:17:15.529299 10539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:15.529392 10539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:15.529434 10539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:15.531304 10539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.532234 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bcf8ba878a74dd09368cd4781d1728d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9477d987e8a042e9ad1e6efcd7a94662"
I20260924 20:17:15.532373 10320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:15.533739 10320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bcf8ba878a74dd09368cd4781d1728d in term 1.
I20260924 20:17:15.534189 10448 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9477d987e8a042e9ad1e6efcd7a94662, 9bcf8ba878a74dd09368cd4781d1728d; no voters:
I20260924 20:17:15.534811 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bcf8ba878a74dd09368cd4781d1728d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba221a077346418ea1d2b48d5c464bd1"
I20260924 20:17:15.534971 10395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:15.536350 10395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bcf8ba878a74dd09368cd4781d1728d in term 1.
I20260924 20:17:15.536808 10539 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [CANDIDATE]: Term 1 election: Requested vote from peers 9477d987e8a042e9ad1e6efcd7a94662 (127.29.83.190:36895), ba221a077346418ea1d2b48d5c464bd1 (127.29.83.189:35007)
I20260924 20:17:15.537022 10539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:15.539264 10539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [term 1 LEADER]: Becoming Leader. State: Replica: 9bcf8ba878a74dd09368cd4781d1728d, State: Running, Role: LEADER
I20260924 20:17:15.539713 10539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } }
I20260924 20:17:15.543085 10539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcf8ba878a74dd09368cd4781d1728d. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.543274 10539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:15.543745 10554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:15.544792 10554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:15.552098 10395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [term 1 FOLLOWER]: Refusing update from remote peer 9bcf8ba878a74dd09368cd4781d1728d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:15.552961 10552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:15.558709 10506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcf8ba878a74dd09368cd4781d1728d. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.558852 10506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.561569 10320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [term 1 FOLLOWER]: Refusing update from remote peer 9bcf8ba878a74dd09368cd4781d1728d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:15.562335 10552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:15.567013 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcf8ba878a74dd09368cd4781d1728d. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.567169 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.578181 10552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.578374 10552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:15.579912 10506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.580040 10506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.580538 10555 mvcc.cc:205] Tried to move back new op lower bound from 7332991121613291520 to 7332991121575342080. Current Snapshot: MvccSnapshot[applied={T|T < 7332991121613291520}]
I20260924 20:17:15.581646 10539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.581799 10539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:15.582926 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bcf8ba878a74dd09368cd4781d1728d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9477d987e8a042e9ad1e6efcd7a94662" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36895 } } peers { permanent_uuid: "ba221a077346418ea1d2b48d5c464bd1" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 35007 } } peers { permanent_uuid: "9bcf8ba878a74dd09368cd4781d1728d" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46103 } } }
I20260924 20:17:15.583050 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:17:15.583874 10556 mvcc.cc:205] Tried to move back new op lower bound from 7332991121613291520 to 7332991121575342080. Current Snapshot: MvccSnapshot[applied={T|T < 7332991121613291520}]
I20260924 20:17:15.585171 10558 mvcc.cc:205] Tried to move back new op lower bound from 7332991121613291520 to 7332991121575342080. Current Snapshot: MvccSnapshot[applied={T|T < 7332991121613291520}]
I20260924 20:17:15.592201 10554 catalog_manager.cc:1347] Generated new cluster ID: bc5465ebe34c43669de3c2726a8cf7dc
I20260924 20:17:15.592294 10554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:15.629920 10554 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:17:15.631462 10554 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:15.646634 10554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcf8ba878a74dd09368cd4781d1728d: Generated new TSK 0
I20260924 20:17:15.647581 10554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:17:16.442628 10528 catalog_manager.cc:1259] Loaded cluster ID: bc5465ebe34c43669de3c2726a8cf7dc
I20260924 20:17:16.442643 10523 catalog_manager.cc:1259] Loaded cluster ID: bc5465ebe34c43669de3c2726a8cf7dc
I20260924 20:17:16.442749 10523 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: loading cluster ID for follower catalog manager: success
I20260924 20:17:16.442749 10528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: loading cluster ID for follower catalog manager: success
I20260924 20:17:16.444381 10528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: acquiring CA information for follower catalog manager: success
I20260924 20:17:16.444383 10523 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: acquiring CA information for follower catalog manager: success
I20260924 20:17:16.445576 10528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba221a077346418ea1d2b48d5c464bd1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:17:16.445576 10523 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9477d987e8a042e9ad1e6efcd7a94662: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:17:16.720750 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10278
I20260924 20:17:16.727289 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10353
I20260924 20:17:16.733340 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10429
2026-09-24T20:17:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3165 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:17:16 dist-test-slave-52fr krb5kdc[10569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:17:16 dist-test-slave-52fr krb5kdc[10569](info): set up 2 sockets
Sep 24 20:17:16 dist-test-slave-52fr krb5kdc[10569](info): commencing operation
krb5kdc: starting...
W20260924 20:17:17.777937 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:17:17 dist-test-slave-52fr krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:17:17.902948 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34989
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:18.019685 10582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:18.019976 10582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:18.020020 10582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:18.024194 10582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:18.024279 10582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:18.024302 10582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:18.024318 10582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:18.024340 10582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:18.030830 10582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:18.032017 10582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:18.033110 10582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:18.042191 10587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:18.042203 10588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:18.042191 10590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:18.042599 10582 server_base.cc:1034] running on GCE node
I20260924 20:17:18.043175 10582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:18.044275 10582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:18.045465 10582 hybrid_clock.cc:648] HybridClock initialized: now 1790281038045418 us; error 66 us; skew 500 ppm
Sep 24 20:17:18 dist-test-slave-52fr krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281038, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:18.048453 10582 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:18.049543 10582 webserver.cc:460] Webserver started at http://127.29.83.190:34619/ using document root <none> and password file <none>
I20260924 20:17:18.050211 10582 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:18.050311 10582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:18.050549 10582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:18.052697 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "28e025ab89e74c62be3443b388c229e1"
format_stamp: "Formatted at 2026-09-24 20:17:18 on dist-test-slave-52fr"
server_key: "69bb7461dabcd3fa9af76a38dd30dd33"
server_key_iv: "74e71f9d8bb9f2d323464472efd2d900"
server_key_version: "encryptionkey@0"
I20260924 20:17:18.053318 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "28e025ab89e74c62be3443b388c229e1"
format_stamp: "Formatted at 2026-09-24 20:17:18 on dist-test-slave-52fr"
server_key: "69bb7461dabcd3fa9af76a38dd30dd33"
server_key_iv: "74e71f9d8bb9f2d323464472efd2d900"
server_key_version: "encryptionkey@0"
I20260924 20:17:18.057279 10582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:17:18.060062 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:18.061395 10582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:18.061566 10582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "28e025ab89e74c62be3443b388c229e1"
format_stamp: "Formatted at 2026-09-24 20:17:18 on dist-test-slave-52fr"
server_key: "69bb7461dabcd3fa9af76a38dd30dd33"
server_key_iv: "74e71f9d8bb9f2d323464472efd2d900"
server_key_version: "encryptionkey@0"
I20260924 20:17:18.061698 10582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:18.078866 10582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:18.082286 10582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:18.082516 10582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:18.091161 10582 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:34989
I20260924 20:17:18.091236 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:34989 every 8 connection(s)
I20260924 20:17:18.092411 10582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:18.095912 10650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:18.100118 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10582
I20260924 20:17:18.100312 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:17:18.100597 30030 external_mini_cluster.cc:1428] Setting key 43915e4bf096f9d0b0dd4012f71af719
I20260924 20:17:18.103204 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: Bootstrap starting.
I20260924 20:17:18.106063 10650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:18.106976 10650 log.cc:824] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:18.109683 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: No bootstrap required, opened a new log
Sep 24 20:17:18 dist-test-slave-52fr krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:18.113449 10650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } }
I20260924 20:17:18.113730 10650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:18.113878 10650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e025ab89e74c62be3443b388c229e1, State: Initialized, Role: FOLLOWER
I20260924 20:17:18.114383 10650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } }
I20260924 20:17:18.114543 10650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:18.114635 10650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:18.114750 10650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:18.115713 10650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } }
I20260924 20:17:18.116091 10650 leader_election.cc:302] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e025ab89e74c62be3443b388c229e1; no voters:
I20260924 20:17:18.116437 10650 leader_election.cc:288] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:17:18.116561 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:18.116787 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [term 1 LEADER]: Becoming Leader. State: Replica: 28e025ab89e74c62be3443b388c229e1, State: Running, Role: LEADER
I20260924 20:17:18.117185 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } }
I20260924 20:17:18.117676 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:18.102220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.29.83.190:34989)
0924 20:17:18.102670 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:18.102686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:18.102724 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:17:18.103554 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:18.103577 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:18.103967 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:18.104267 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:18.105241 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:18.106470 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:18.107668 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:18.108304 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:18.108453 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:18.111506 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:18.111511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:18.111526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:18.111555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:18.114119 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:18.114664 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:18.114670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:18.114676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:18.114863 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:18.115142 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:18.115147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:18.115149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:18.115579 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:18.115809 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:18.116104 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 20:17:18.116622 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260924 20:17:18.117810 10650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:18.119870 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28e025ab89e74c62be3443b388c229e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } } }
I20260924 20:17:18.120225 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28e025ab89e74c62be3443b388c229e1. Latest consensus state: current_term: 1 leader_uuid: "28e025ab89e74c62be3443b388c229e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e025ab89e74c62be3443b388c229e1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34989 } } }
I20260924 20:17:18.120007 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:18.120522 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:18.121423 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 20:17:18.124451 10669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:17:18.124557 10669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:17:18.124665 10670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:18.125499 10670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:18.131621 10670 catalog_manager.cc:1347] Generated new cluster ID: fb45134711084fd3839c79495705a700
I20260924 20:17:18.131708 10670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:18.152981 10670 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:17:18.154408 10670 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:18.160310 10670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28e025ab89e74c62be3443b388c229e1: Generated new TSK 0
I20260924 20:17:18.161084 10670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:17:18.177933 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10582
2026-09-24T20:17:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1443 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:17:18 dist-test-slave-52fr krb5kdc[10677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:17:18 dist-test-slave-52fr krb5kdc[10677](info): set up 2 sockets
Sep 24 20:17:18 dist-test-slave-52fr krb5kdc[10677](info): commencing operation
krb5kdc: starting...
W20260924 20:17:19.230157 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:17:19 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:17:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:17:19.361346 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:19.473356 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:19.473688 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:19.473793 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:19.478045 10690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:19.478145 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:19.478183 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:19.478220 10690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:19.478255 10690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:19.483881 10690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:19.485066 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:19.486217 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:19.495214 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:19.495170 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:19.495170 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:19.495628 10690 server_base.cc:1034] running on GCE node
I20260924 20:17:19.496366 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:19.497694 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:19.498924 10690 hybrid_clock.cc:648] HybridClock initialized: now 1790281039498906 us; error 33 us; skew 500 ppm
Sep 24 20:17:19 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281039, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:19.501868 10690 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:19.502758 10690 webserver.cc:460] Webserver started at http://127.29.83.190:43449/ using document root <none> and password file <none>
I20260924 20:17:19.503334 10690 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:19.503440 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:19.503684 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:19.505843 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "45b051c9dfea47b1a747cc7da7407d85"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "b55db557e82db4234b67bc5831686676"
server_key_iv: "35a8e48fedad6a6f5b49b7333ded1507"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.506474 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "45b051c9dfea47b1a747cc7da7407d85"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "b55db557e82db4234b67bc5831686676"
server_key_iv: "35a8e48fedad6a6f5b49b7333ded1507"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.510679 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:17:19.513492 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:19.515048 10690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:19.515260 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "45b051c9dfea47b1a747cc7da7407d85"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "b55db557e82db4234b67bc5831686676"
server_key_iv: "35a8e48fedad6a6f5b49b7333ded1507"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.515414 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:19.527842 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:19.530961 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:19.531181 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:19.538728 10690 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38959
I20260924 20:17:19.539034 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38959 every 8 connection(s)
I20260924 20:17:19.539770 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:19.542754 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:19.548164 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10690
I20260924 20:17:19.548179 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap starting.
I20260924 20:17:19.548341 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:17:19.548609 30030 external_mini_cluster.cc:1428] Setting key 9f779f7dc2079e09614d96721b424c5c
I20260924 20:17:19.550771 10758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:19.551506 10758 log.cc:824] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:19.553478 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: No bootstrap required, opened a new log
I20260924 20:17:19.557339 10758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:19.557605 10758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:19.557689 10758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Initialized, Role: FOLLOWER
Sep 24 20:17:19 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:19.558256 10758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:19.558429 10758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:19.558518 10758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:19.558615 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:19.559859 10758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:19.560276 10758 leader_election.cc:302] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b051c9dfea47b1a747cc7da7407d85; no voters:
I20260924 20:17:19.560640 10758 leader_election.cc:288] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:17:19.560722 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:19.560971 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 LEADER]: Becoming Leader. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Running, Role: LEADER
I20260924 20:17:19.561280 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:19.562045 10758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:19.563099 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b051c9dfea47b1a747cc7da7407d85. Latest consensus state: current_term: 1 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:19.563230 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:19.563537 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:19.563637 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:19.563642 10772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:19.565732 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:19.550146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.29.83.190:38959)
0924 20:17:19.550568 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:19.550584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:19.550618 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:17:19.551307 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:19.551334 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:19.551725 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:19.552072 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:19.553030 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.554291 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.555275 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.555770 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.555953 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:19.558810 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:19.558815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:19.558832 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:19.558860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:19.561326 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:19.562028 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:19.562034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:19.562041 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:19.562192 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:19.562551 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:19.562556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:19.562559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:19.563228 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:19.563450 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:19.564430 (+ 980us) server_negotiation.cc:299] Negotiation successful
0924 20:17:19.564750 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260924 20:17:19.567063 10772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:19.573940 10772 catalog_manager.cc:1347] Generated new cluster ID: b3848e7c62694eee9f5d4c041e65b56c
I20260924 20:17:19.574043 10772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:19.585182 10772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:17:19.586272 10772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:19.596539 10772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Generated new TSK 0
I20260924 20:17:19.597563 10772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:17:19.633478 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:19.746884 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:19.747205 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:19.747291 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:19.751426 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:19.751519 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:19.751619 10784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:19.756877 10784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:19.758069 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:19.759152 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:19.769536 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:19.769591 10784 server_base.cc:1034] running on GCE node
W20260924 20:17:19.769510 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:19.769516 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:19.770349 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:19.771071 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:19.772266 10784 hybrid_clock.cc:648] HybridClock initialized: now 1790281039772226 us; error 54 us; skew 500 ppm
Sep 24 20:17:19 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281039, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:19.775241 10784 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:19.776217 10784 webserver.cc:460] Webserver started at http://127.29.83.129:34355/ using document root <none> and password file <none>
I20260924 20:17:19.776813 10784 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:19.776911 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:19.777150 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:19.779318 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "076c59b72cbc438daaca427a3f570f64"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "2f8948c727a54cdd4b28c66f8672fc74"
server_key_iv: "fd7d6e36537a494d43442238e2489207"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.779901 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "076c59b72cbc438daaca427a3f570f64"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "2f8948c727a54cdd4b28c66f8672fc74"
server_key_iv: "fd7d6e36537a494d43442238e2489207"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.783881 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:17:19.786443 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:19.787671 10784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:17:19.787810 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "076c59b72cbc438daaca427a3f570f64"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "2f8948c727a54cdd4b28c66f8672fc74"
server_key_iv: "fd7d6e36537a494d43442238e2489207"
server_key_version: "encryptionkey@0"
I20260924 20:17:19.787930 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:19.813585 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:19.817337 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:19.817562 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:19.818295 10784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:19.819231 10784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:19.819298 10784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:19.819366 10784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:19.819414 10784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:19.828574 10784 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:37675
I20260924 20:17:19.828596 10912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:37675 every 8 connection(s)
I20260924 20:17:19.829608 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:19.831001 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10784
I20260924 20:17:19.831147 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:17:19.831431 30030 external_mini_cluster.cc:1428] Setting key 05a362ed0d8f66f76102ec45ac58d65e
Sep 24 20:17:19 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281039, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
I20260924 20:17:19.845180 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:19.831808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:59131 (local address 127.29.83.190:38959)
0924 20:17:19.831975 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:19.831982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:19.833010 (+ 1028us) server_negotiation.cc:407] Connection header received
0924 20:17:19.834574 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:19.834578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:19.834659 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:19.834731 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:19.836528 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.837076 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.838348 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.838610 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.838730 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:19.841502 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:19.841505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:19.841507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:19.841532 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:19.843023 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:19.843566 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:19.843569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:19.843571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:19.843649 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:19.843991 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:19.843995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:19.843997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:19.844184 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:19.844286 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:19.844874 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 20:17:19.845007 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:17:19.846165 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:19.832292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:59131)
0924 20:17:19.832847 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:19.832911 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:19.834277 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:19.835016 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:19.835027 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:19.835803 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:19.836371 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:19.836386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.837224 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.837228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:19.838221 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:19.838232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:19.838777 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:19.838781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:19.838957 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:19.839584 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:19.839606 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:19.841312 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:19.843167 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:19.843173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:19.843195 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:19.843445 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:19.843741 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:19.843746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:19.843748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:19.843893 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:19.844299 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:19.844307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:19.844614 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 20:17:19.844834 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 20:17:19.845199 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260924 20:17:19.847638 10913 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:19.847914 10913 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:19.848526 10913 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:19.850481 10722 ts_manager.cc:194] Registered new tserver with Master: 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675)
I20260924 20:17:19.851687 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:59131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:17:19.863301 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:19.980661 10918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:19.980983 10918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:19.981079 10918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:19.985186 10918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:19.985285 10918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:19.985385 10918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:19.990796 10918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:19.991940 10918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:19.993029 10918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:20.003142 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.003144 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.003142 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:20.003427 10918 server_base.cc:1034] running on GCE node
I20260924 20:17:20.003919 10918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:20.004591 10918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:20.005832 10918 hybrid_clock.cc:648] HybridClock initialized: now 1790281040005731 us; error 45 us; skew 500 ppm
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:20.008637 10918 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:20.009732 10918 webserver.cc:460] Webserver started at http://127.29.83.130:40665/ using document root <none> and password file <none>
I20260924 20:17:20.010339 10918 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:20.010408 10918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:20.010573 10918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:20.012581 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "c82210f4e0a7489fac84d5034c5b3d88"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "b53dcbfd58728bbfff56bbcbe4ff4a8f"
server_key_iv: "484d7c46cf4adeeedbee5782dddba73a"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.013130 10918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "c82210f4e0a7489fac84d5034c5b3d88"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "b53dcbfd58728bbfff56bbcbe4ff4a8f"
server_key_iv: "484d7c46cf4adeeedbee5782dddba73a"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.016937 10918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:17:20.019380 10933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.020720 10918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:17:20.020843 10918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "c82210f4e0a7489fac84d5034c5b3d88"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "b53dcbfd58728bbfff56bbcbe4ff4a8f"
server_key_iv: "484d7c46cf4adeeedbee5782dddba73a"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.020955 10918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:20.040936 10918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:20.044129 10918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:20.044337 10918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:20.044939 10918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:20.045809 10918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:20.045876 10918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.045943 10918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:20.045989 10918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.055286 10918 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:35363
I20260924 20:17:20.055363 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:35363 every 8 connection(s)
I20260924 20:17:20.056210 10918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:20.060447 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 10918
I20260924 20:17:20.060602 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:17:20.060892 30030 external_mini_cluster.cc:1428] Setting key 9f17e1d77258a195d57c91e1ced560a5
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:17:20.072629 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.058179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:34651 (local address 127.29.83.190:38959)
0924 20:17:20.058323 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.058327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.059225 (+ 898us) server_negotiation.cc:407] Connection header received
0924 20:17:20.060620 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.060623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.060683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.060765 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:20.062057 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.062768 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.064173 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.064566 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.064686 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.067702 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:20.067709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:20.067715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:20.067753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:20.069842 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.070447 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.070451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.070453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.070550 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.071134 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.071137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.071139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.071401 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:20.071514 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.072249 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.072469 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:17:20.073539 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.058576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:34651)
0924 20:17:20.059041 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.059091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.060401 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.060937 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.060943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.061346 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:20.061900 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.061915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.062924 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.062927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.064041 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.064052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.064769 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.064775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.064999 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.065795 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:20.065819 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:20.067518 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:20.070014 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.070020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.070035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.070329 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.070824 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.070826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.070828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.071013 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.071535 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:20.071541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:20.071984 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.072233 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.072588 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:17:20.074888 11047 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:20.075156 11047 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:20.075681 11047 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:20.076750 10722 ts_manager.cc:194] Registered new tserver with Master: c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:20.077417 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:34651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:17:20.094224 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:20.206100 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:20.206405 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:20.206451 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:20.210623 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:20.210692 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:20.210753 11052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:20.216154 11052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:20.217238 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:20.218366 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:20.228745 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.228729 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.228765 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:20.229166 11052 server_base.cc:1034] running on GCE node
I20260924 20:17:20.230144 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:20.230851 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:20.232071 11052 hybrid_clock.cc:648] HybridClock initialized: now 1790281040232026 us; error 61 us; skew 500 ppm
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:20.235132 11052 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:20.236064 11052 webserver.cc:460] Webserver started at http://127.29.83.131:46611/ using document root <none> and password file <none>
I20260924 20:17:20.236668 11052 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:20.236763 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:20.236980 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:20.239154 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "01da3a139bbc409aad6a9846818265bf"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "477ead512acc5b4cd9496d314b69ddb4"
server_key_iv: "c67e2198821208dd62a06156f966487d"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.239733 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "01da3a139bbc409aad6a9846818265bf"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "477ead512acc5b4cd9496d314b69ddb4"
server_key_iv: "c67e2198821208dd62a06156f966487d"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.243691 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:17:20.246472 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.247831 11052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:20.247980 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "01da3a139bbc409aad6a9846818265bf"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "477ead512acc5b4cd9496d314b69ddb4"
server_key_iv: "c67e2198821208dd62a06156f966487d"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.248106 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:20.269289 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:20.272557 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:20.272799 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:20.273439 11052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:20.274341 11052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:20.274415 11052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.274482 11052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:20.274526 11052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.283357 11052 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42607
I20260924 20:17:20.283375 11180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42607 every 8 connection(s)
I20260924 20:17:20.284289 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:20.291208 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11052
I20260924 20:17:20.291356 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:17:20.291657 30030 external_mini_cluster.cc:1428] Setting key 6d54877b00e67166f363471b6143f79e
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:20.298913 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.286149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:41277 (local address 127.29.83.190:38959)
0924 20:17:20.286299 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.286302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.287274 (+ 972us) server_negotiation.cc:407] Connection header received
0924 20:17:20.288637 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.288640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.288693 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.288773 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:20.290069 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.290597 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.292292 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.292603 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.292706 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.295428 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:20.295432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:20.295434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:20.295468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:20.296774 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.297310 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.297322 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.297324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.297407 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.297786 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.297789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.297791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.297988 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:20.298102 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.298620 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.298743 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:17:20.299754 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.286593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:41277)
0924 20:17:20.287087 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.287136 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.288420 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.288964 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.288971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.289338 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:20.289926 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.289944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.290703 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.290705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.292174 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.292184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.292768 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.292772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.292976 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.293629 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:20.293649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:20.295260 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:20.296900 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.296907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.296926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.297202 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.297505 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.297509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.297511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.297661 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.298101 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:20.298108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:20.298372 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.298565 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.298930 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260924 20:17:20.300886 11181 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:20.301139 11181 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:20.301621 11181 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:20.302647 10722 ts_manager.cc:194] Registered new tserver with Master: 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607)
I20260924 20:17:20.303208 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:41277
I20260924 20:17:20.306129 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:17:20.315730 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.308224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.29.83.190:38959)
0924 20:17:20.308362 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.308366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.308662 (+ 296us) server_negotiation.cc:407] Connection header received
0924 20:17:20.308810 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.308812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.308855 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.308935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:20.309644 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.310285 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.311112 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.311311 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.311399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.312862 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:20.312865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:20.312867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:20.312890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:20.314255 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.314661 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.314664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.314665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.314738 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.314992 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.314994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.314995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.315199 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:20.315300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.315512 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.315609 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:17:20.319051 10722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:17:20.322367 10722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:17:20.337885 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.332221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.0.0.1:59398)
0924 20:17:20.333370 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.333383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.333477 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.333936 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.333939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.333958 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.334035 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.334040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.335333 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.335337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.336449 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.336459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337314 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.337318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.337418 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.337453 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.337565 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.337687 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":45,"thread_start_us":163,"threads_started":1}
I20260924 20:17:20.337895 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.332408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.29.83.130:35363)
0924 20:17:20.333230 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.333241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.333485 (+ 244us) server_negotiation.cc:407] Connection header received
0924 20:17:20.333583 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.333588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.333814 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.333961 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.334114 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.335175 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.336593 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337177 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.337323 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.337478 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.337593 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.337687 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":104,"threads_started":1}
I20260924 20:17:20.338193 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.331485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42607 (local address 127.0.0.1:41554)
0924 20:17:20.332038 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.332089 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.332218 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.333641 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.333644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.333661 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.333784 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.333792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.334822 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.334827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.336458 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.336470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337590 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.337594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.337712 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.337733 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.337820 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.337901 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":110,"thread_start_us":117,"threads_started":1}
I20260924 20:17:20.338892 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.331490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.0.0.1:37748)
0924 20:17:20.332414 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.332427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.332516 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.333203 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.333209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.333232 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.333385 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.333393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.334933 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.334936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.336458 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.336470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337590 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.337594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.338489 (+ 895us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.338513 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.338611 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.338700 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":806,"thread_start_us":87,"threads_started":1}
I20260924 20:17:20.339008 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.331571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.29.83.129:37675)
0924 20:17:20.331875 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.331896 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.332532 (+ 636us) server_negotiation.cc:407] Connection header received
0924 20:17:20.332783 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.332790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.333027 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.333213 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.333538 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.334762 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.336606 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337443 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.337640 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.337830 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.338665 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.338821 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260924 20:17:20.339612 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.332746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.29.83.131:42607)
0924 20:17:20.333121 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.333133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.333156 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:17:20.333231 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.333238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.333457 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.333671 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.333883 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.334692 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.336607 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.337443 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.339084 (+ 1641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.339242 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.339297 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.339428 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":151,"threads_started":1}
I20260924 20:17:20.341401 10981 tablet_service.cc:1467] Processing CreateTablet for tablet 6de002dee89e472b86708741a9243b2f (DEFAULT_TABLE table=test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:20.341671 10847 tablet_service.cc:1467] Processing CreateTablet for tablet 6de002dee89e472b86708741a9243b2f (DEFAULT_TABLE table=test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:20.342592 10847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de002dee89e472b86708741a9243b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:20.342599 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de002dee89e472b86708741a9243b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:20.342325 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 6de002dee89e472b86708741a9243b2f (DEFAULT_TABLE table=test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:20.343231 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de002dee89e472b86708741a9243b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:20.348892 11198 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Bootstrap starting.
I20260924 20:17:20.350374 11199 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Bootstrap starting.
I20260924 20:17:20.351397 11198 tablet_bootstrap.cc:654] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:20.352330 11198 log.cc:824] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:20.353117 11199 tablet_bootstrap.cc:654] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:20.353181 11200 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Bootstrap starting.
I20260924 20:17:20.354149 11199 log.cc:824] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:20.354877 11198 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: No bootstrap required, opened a new log
I20260924 20:17:20.355108 11198 ts_tablet_manager.cc:1397] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260924 20:17:20.355376 11200 tablet_bootstrap.cc:654] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:20.356277 11199 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: No bootstrap required, opened a new log
I20260924 20:17:20.356475 11199 ts_tablet_manager.cc:1397] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:17:20.356719 11200 log.cc:824] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:20.358814 11200 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: No bootstrap required, opened a new log
I20260924 20:17:20.359067 11200 ts_tablet_manager.cc:1397] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:17:20.359071 11198 raft_consensus.cc:348] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.359366 11198 raft_consensus.cc:374] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:20.359447 11198 raft_consensus.cc:729] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076c59b72cbc438daaca427a3f570f64, State: Initialized, Role: FOLLOWER
I20260924 20:17:20.359887 11198 consensus_queue.cc:260] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.360553 10913 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
I20260924 20:17:20.360885 11198 ts_tablet_manager.cc:1428] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:17:20.360281 11199 raft_consensus.cc:348] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.361235 11199 raft_consensus.cc:374] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:20.361333 11199 raft_consensus.cc:729] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82210f4e0a7489fac84d5034c5b3d88, State: Initialized, Role: FOLLOWER
I20260924 20:17:20.361959 11199 consensus_queue.cc:260] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.362618 11047 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
I20260924 20:17:20.363078 11199 ts_tablet_manager.cc:1428] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:17:20.363040 11200 raft_consensus.cc:348] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.363278 11200 raft_consensus.cc:374] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:20.363339 11200 raft_consensus.cc:729] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01da3a139bbc409aad6a9846818265bf, State: Initialized, Role: FOLLOWER
I20260924 20:17:20.363756 11200 consensus_queue.cc:260] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.364418 11181 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
I20260924 20:17:20.364771 11200 ts_tablet_manager.cc:1428] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:17:20.457340 11206 raft_consensus.cc:482] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:20.457512 11204 raft_consensus.cc:482] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:20.457571 11206 raft_consensus.cc:504] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.457798 11204 raft_consensus.cc:504] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.459076 11204 leader_election.cc:288] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:20.459880 11206 leader_election.cc:288] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:20.463917 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.459278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.29.83.131:41133)
0924 20:17:20.459478 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.459495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.459583 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.459946 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.459952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.459971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.460102 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.460109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.461093 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.461096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.462345 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.462355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.463331 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.463529 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.463554 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.463628 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.463711 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":30,"spinlock_wait_cycles":1920}
I20260924 20:17:20.464380 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.460012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:42109 (local address 127.29.83.130:35363)
0924 20:17:20.460133 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.460137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.460206 (+ 69us) server_negotiation.cc:407] Connection header received
0924 20:17:20.460382 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.460386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.460435 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.460485 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.460841 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.461543 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.462987 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.463630 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463732 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.463779 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.464156 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.464232 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:17:20.464701 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.459644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.29.83.129:42109)
0924 20:17:20.460109 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.460122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.460267 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.460604 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.460606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.460620 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.460713 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.460717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.461679 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.461682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.462855 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.462863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.463836 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.464005 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.464042 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.464512 (+ 470us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.464575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260924 20:17:20.464846 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.459368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:37075 (local address 127.29.83.131:42607)
0924 20:17:20.459720 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.459725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.459737 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:17:20.459874 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.459879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.459925 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.459972 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.460426 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.461908 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463308 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.463880 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463960 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.464005 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.464633 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.464701 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":750,"server-negotiator.queue_time_us":296}
I20260924 20:17:20.464586 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.459263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42607 (local address 127.29.83.129:37075)
0924 20:17:20.459599 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.459611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.459734 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.460122 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.460125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.460143 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.460297 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.460306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.461998 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.462004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.463184 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.463193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.464153 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.464158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.464249 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.464271 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.464337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.464414 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":88,"spinlock_wait_cycles":8320,"thread_start_us":104,"threads_started":1}
I20260924 20:17:20.465392 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "076c59b72cbc438daaca427a3f570f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82210f4e0a7489fac84d5034c5b3d88" is_pre_election: true
I20260924 20:17:20.465737 11001 raft_consensus.cc:2454] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 076c59b72cbc438daaca427a3f570f64 in term 0.
I20260924 20:17:20.466279 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.459418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:41133 (local address 127.29.83.129:37675)
0924 20:17:20.459543 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.459549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.459563 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:17:20.459670 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.459673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.459748 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.459820 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.460234 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.460983 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.462475 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.463098 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.463213 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.463269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.464352 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.464739 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":18816}
I20260924 20:17:20.466435 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "076c59b72cbc438daaca427a3f570f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01da3a139bbc409aad6a9846818265bf" is_pre_election: true
I20260924 20:17:20.466663 11135 raft_consensus.cc:2454] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 076c59b72cbc438daaca427a3f570f64 in term 0.
I20260924 20:17:20.466822 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "01da3a139bbc409aad6a9846818265bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "076c59b72cbc438daaca427a3f570f64" is_pre_election: true
I20260924 20:17:20.466907 10800 leader_election.cc:302] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64, c82210f4e0a7489fac84d5034c5b3d88; no voters:
I20260924 20:17:20.467010 10867 raft_consensus.cc:2454] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01da3a139bbc409aad6a9846818265bf in term 0.
I20260924 20:17:20.467329 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.460439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.29.83.131:38707)
0924 20:17:20.461055 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.461068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.461157 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.463420 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.463422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.463439 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:20.463639 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.463645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.464631 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.464634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.465812 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.465823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.466882 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.466887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.466991 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.467015 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.467092 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.467167 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":55,"thread_start_us":98,"threads_started":1}
I20260924 20:17:20.467401 11070 leader_election.cc:302] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01da3a139bbc409aad6a9846818265bf, 076c59b72cbc438daaca427a3f570f64; no voters:
I20260924 20:17:20.467495 11204 raft_consensus.cc:2781] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:20.467590 11204 raft_consensus.cc:482] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:20.467648 11204 raft_consensus.cc:3037] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:20.467710 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.460780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:38707 (local address 127.29.83.130:35363)
0924 20:17:20.463118 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.463130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.463145 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:17:20.463190 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.463192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.463250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.463325 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:20.463766 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.464511 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.465971 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.466622 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.466767 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.466821 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.467485 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.467561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2262,"thread_start_us":76,"threads_started":1}
I20260924 20:17:20.467782 11206 raft_consensus.cc:2781] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:20.467859 11206 raft_consensus.cc:482] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:20.467916 11206 raft_consensus.cc:3037] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:20.468046 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "01da3a139bbc409aad6a9846818265bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82210f4e0a7489fac84d5034c5b3d88" is_pre_election: true
I20260924 20:17:20.468171 11001 raft_consensus.cc:2454] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01da3a139bbc409aad6a9846818265bf in term 0.
I20260924 20:17:20.469086 11206 raft_consensus.cc:504] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.469172 11204 raft_consensus.cc:504] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.469532 11206 leader_election.cc:288] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [CANDIDATE]: Term 1 election: Requested vote from peers 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:20.469584 11204 leader_election.cc:288] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 1 election: Requested vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:20.469863 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "01da3a139bbc409aad6a9846818265bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "076c59b72cbc438daaca427a3f570f64"
I20260924 20:17:20.469952 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "076c59b72cbc438daaca427a3f570f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01da3a139bbc409aad6a9846818265bf"
I20260924 20:17:20.469995 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "076c59b72cbc438daaca427a3f570f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82210f4e0a7489fac84d5034c5b3d88"
I20260924 20:17:20.470048 10867 raft_consensus.cc:2379] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 01da3a139bbc409aad6a9846818265bf in current term 1: Already voted for candidate 076c59b72cbc438daaca427a3f570f64 in this term.
I20260924 20:17:20.470130 11135 raft_consensus.cc:2379] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 076c59b72cbc438daaca427a3f570f64 in current term 1: Already voted for candidate 01da3a139bbc409aad6a9846818265bf in this term.
I20260924 20:17:20.470108 11001 raft_consensus.cc:3037] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:20.470091 11000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de002dee89e472b86708741a9243b2f" candidate_uuid: "01da3a139bbc409aad6a9846818265bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82210f4e0a7489fac84d5034c5b3d88"
I20260924 20:17:20.470706 11068 leader_election.cc:302] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01da3a139bbc409aad6a9846818265bf; no voters: 076c59b72cbc438daaca427a3f570f64, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:20.470873 11206 raft_consensus.cc:2726] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 20:17:20.471422 11001 raft_consensus.cc:2454] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 076c59b72cbc438daaca427a3f570f64 in term 1.
I20260924 20:17:20.471719 10800 leader_election.cc:302] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64, c82210f4e0a7489fac84d5034c5b3d88; no voters: 01da3a139bbc409aad6a9846818265bf
I20260924 20:17:20.471938 11204 raft_consensus.cc:2781] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:20.472153 11204 raft_consensus.cc:686] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 LEADER]: Becoming Leader. State: Replica: 076c59b72cbc438daaca427a3f570f64, State: Running, Role: LEADER
I20260924 20:17:20.472520 11204 consensus_queue.cc:237] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.476257 10722 catalog_manager.cc:5495] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 076c59b72cbc438daaca427a3f570f64 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "076c59b72cbc438daaca427a3f570f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } health_report { overall_health: UNKNOWN } } }
I20260924 20:17:20.497495 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.494324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.29.83.129:37675)
0924 20:17:20.494447 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.494450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.494463 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:17:20.494540 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.494542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.494595 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.494698 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:17:20.494928 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.495422 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.496226 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.496436 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.496514 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.496762 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:17:20.497198 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:17:20.497288 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.497341 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.497385 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:17:20.503592 11135 raft_consensus.cc:1261] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Refusing update from remote peer 076c59b72cbc438daaca427a3f570f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:20.503592 11001 raft_consensus.cc:1261] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Refusing update from remote peer 076c59b72cbc438daaca427a3f570f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:20.504333 11204 consensus_queue.cc:1035] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:20.504609 11211 consensus_queue.cc:1035] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:20.512013 11214 mvcc.cc:205] Tried to move back new op lower bound from 7332991141894651904 to 7332991141779288064. Current Snapshot: MvccSnapshot[applied={T|T < 7332991141894651904}]
I20260924 20:17:20.514019 11216 mvcc.cc:205] Tried to move back new op lower bound from 7332991141894651904 to 7332991141779288064. Current Snapshot: MvccSnapshot[applied={T|T < 7332991141894651904}]
I20260924 20:17:20.522046 10721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:20.522235 10721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:20.524801 10721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Sending DeleteTablet for 3 replicas of tablet 6de002dee89e472b86708741a9243b2f
I20260924 20:17:20.525658 10847 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de002dee89e472b86708741a9243b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:20 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37748
I20260924 20:17:20.525724 11115 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de002dee89e472b86708741a9243b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:20 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:41554
I20260924 20:17:20.525862 10981 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de002dee89e472b86708741a9243b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:20 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:59398
I20260924 20:17:20.526283 11224 tablet_replica.cc:330] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: stopping tablet replica
I20260924 20:17:20.526333 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10784
I20260924 20:17:20.527179 11224 raft_consensus.cc:2229] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:20.527532 11224 raft_consensus.cc:2258] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:20.527913 11223 tablet_replica.cc:330] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: stopping tablet replica
I20260924 20:17:20.528250 11223 raft_consensus.cc:2229] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:20.528568 11223 raft_consensus.cc:2258] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:20.529147 11224 ts_tablet_manager.cc:1905] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:20.530079 11223 ts_tablet_manager.cc:1905] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:20.531752 11224 ts_tablet_manager.cc:1918] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:20.531867 11224 log.cc:1196] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/6de002dee89e472b86708741a9243b2f
I20260924 20:17:20.532179 11224 ts_tablet_manager.cc:1939] T 6de002dee89e472b86708741a9243b2f P 01da3a139bbc409aad6a9846818265bf: Deleting consensus metadata
I20260924 20:17:20.533309 10706 catalog_manager.cc:4841] TS 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): tablet 6de002dee89e472b86708741a9243b2f (table test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]) successfully deleted
I20260924 20:17:20.533409 11223 ts_tablet_manager.cc:1918] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:20.533537 11223 log.cc:1196] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/6de002dee89e472b86708741a9243b2f
I20260924 20:17:20.534217 11223 ts_tablet_manager.cc:1939] T 6de002dee89e472b86708741a9243b2f P c82210f4e0a7489fac84d5034c5b3d88: Deleting consensus metadata
I20260924 20:17:20.535785 10706 catalog_manager.cc:4841] TS c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): tablet 6de002dee89e472b86708741a9243b2f (table test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]) successfully deleted
W20260924 20:17:20.536072 10708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:37675 (error 108)
I20260924 20:17:20.536125 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10918
I20260924 20:17:20.537418 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.537149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.0.0.1:37768)
0924 20:17:20.537272 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.537345 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:17:20.537590 10708 catalog_manager.cc:4584] TS 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6de002dee89e472b86708741a9243b2f: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37675: connect: Connection refused (error 111)
I20260924 20:17:20.542580 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11052
I20260924 20:17:20.548547 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 10690
I20260924 20:17:20.555066 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--webserver_interface=127.29.83.190
--webserver_port=43449
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:20.666447 11227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:20.666745 11227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:20.666936 11227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:20.671089 11227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:20.671183 11227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:20.671223 11227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:20.671262 11227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:20.671294 11227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:20.676853 11227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:20.678076 11227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:20.679157 11227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:20.691711 11234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.691764 11233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:20.691883 11227 server_base.cc:1034] running on GCE node
W20260924 20:17:20.691776 11236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:20.692624 11227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:20.693717 11227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:20.694936 11227 hybrid_clock.cc:648] HybridClock initialized: now 1790281040694908 us; error 38 us; skew 500 ppm
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:20.697625 11227 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:20.698618 11227 webserver.cc:460] Webserver started at http://127.29.83.190:43449/ using document root <none> and password file <none>
I20260924 20:17:20.699209 11227 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:20.699317 11227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:20.703176 11227 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 20:17:20.705410 11243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.706902 11227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:20.707070 11227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "45b051c9dfea47b1a747cc7da7407d85"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "b55db557e82db4234b67bc5831686676"
server_key_iv: "35a8e48fedad6a6f5b49b7333ded1507"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.707713 11227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:20.726696 11227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:20.729600 11227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:20.729871 11227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:20.737250 11227 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38959
I20260924 20:17:20.737551 11295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38959 every 8 connection(s)
I20260924 20:17:20.738502 11227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:20.741683 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11227
I20260924 20:17:20.742210 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:37675
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=34355
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:20.743566 11296 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:17:20.745311 11296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap starting.
I20260924 20:17:20.762624 11296 log.cc:824] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:20.771136 11296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:20.771538 11296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap complete.
I20260924 20:17:20.775652 11296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:20.776017 11296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Initialized, Role: FOLLOWER
I20260924 20:17:20.776530 11296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:20.776649 11296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:20.776738 11296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:20.776849 11296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:17:20.778739 11296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:20.779057 11296 leader_election.cc:302] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b051c9dfea47b1a747cc7da7407d85; no voters:
I20260924 20:17:20.779374 11296 leader_election.cc:288] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:17:20.779455 11300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:17:20.779640 11300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 LEADER]: Becoming Leader. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Running, Role: LEADER
I20260924 20:17:20.779992 11300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:20.780558 11296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:20.780903 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b051c9dfea47b1a747cc7da7407d85. Latest consensus state: current_term: 2 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:20.781096 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:20.781266 11301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:20.781476 11301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:20.781522 11304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:20.786223 11304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:20.786985 11304 catalog_manager.cc:1259] Loaded cluster ID: b3848e7c62694eee9f5d4c041e65b56c
I20260924 20:17:20.787051 11304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:20.788442 11304 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 20:17:20.793280 11304 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 20:17:20.798749 11304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:20.799512 11304 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Loaded TSK: 0
I20260924 20:17:20.800683 11304 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:17:20.858204 11298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:20.858510 11298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:20.858585 11298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:20.862699 11298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:20.862789 11298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:20.862895 11298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:20.868212 11298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:37675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=34355
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:20.869287 11298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:20.870409 11298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:20.881150 11324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.881260 11326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:20.881318 11323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:20.881649 11298 server_base.cc:1034] running on GCE node
I20260924 20:17:20.882105 11298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:20.882802 11298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:20.884003 11298 hybrid_clock.cc:648] HybridClock initialized: now 1790281040883973 us; error 46 us; skew 500 ppm
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:20.887063 11298 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:20.888203 11298 webserver.cc:460] Webserver started at http://127.29.83.129:34355/ using document root <none> and password file <none>
I20260924 20:17:20.888923 11298 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:20.889025 11298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:20.894083 11298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:17:20.896744 11333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:20.898371 11298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:20.898500 11298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "076c59b72cbc438daaca427a3f570f64"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "2f8948c727a54cdd4b28c66f8672fc74"
server_key_iv: "fd7d6e36537a494d43442238e2489207"
server_key_version: "encryptionkey@0"
I20260924 20:17:20.899245 11298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:20.934581 11298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:20.938359 11298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:20.938558 11298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:20.939222 11298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:20.940551 11340 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:17:20.945029 11298 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:17:20.945097 11298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 20:17:20.945222 11298 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:17:20.946991 11298 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:17:20.947041 11298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 20:17:20.947388 11340 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Bootstrap starting.
I20260924 20:17:20.957466 11298 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:37675
I20260924 20:17:20.957638 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:37675 every 8 connection(s)
I20260924 20:17:20.958523 11298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:20.959790 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11298
I20260924 20:17:20.960325 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:35363
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=40665
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:20.968755 11340 log.cc:824] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:20.976367 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.960604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:50075 (local address 127.29.83.190:38959)
0924 20:17:20.961025 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.961045 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.961488 (+ 443us) server_negotiation.cc:407] Connection header received
0924 20:17:20.963246 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.963264 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.963719 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.964156 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:20.965401 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.966531 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.967779 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.968255 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.968461 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.971173 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:20.971178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:20.971190 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:20.971220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:20.972937 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.973517 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.973523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.973534 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.973652 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:20.974057 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:20.974066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:20.974067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:20.974531 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:20.974675 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:20.975220 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 20:17:20.975492 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
I20260924 20:17:20.976387 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.960614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:50075)
0924 20:17:20.961321 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.961376 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.962662 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.964221 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.964228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.964608 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:20.965213 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.965229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.966720 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.966723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.967601 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.967610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.968425 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.968429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.968624 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.969318 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:20.969341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:20.970965 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:20.973101 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.973107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.973120 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.973394 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.973797 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:20.973800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:20.973802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:20.973946 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:20.974657 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:20.974663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:20.974904 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 20:17:20.975087 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 20:17:20.975446 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":178,"threads_started":1}
I20260924 20:17:20.978680 11340 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:20.979060 11448 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:20.979175 11340 tablet_bootstrap.cc:492] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Bootstrap complete.
I20260924 20:17:20.979332 11448 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:20.979888 11448 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:20.980636 11340 ts_tablet_manager.cc:1397] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260924 20:17:20.982626 11260 ts_manager.cc:194] Registered new tserver with Master: 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675)
I20260924 20:17:20.983934 11340 raft_consensus.cc:348] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.984479 11340 raft_consensus.cc:729] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076c59b72cbc438daaca427a3f570f64, State: Initialized, Role: FOLLOWER
I20260924 20:17:20.984970 11340 consensus_queue.cc:260] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } } peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } }
I20260924 20:17:20.986120 11340 ts_tablet_manager.cc:1428] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260924 20:17:20.987494 11260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:50075
I20260924 20:17:20.990221 11448 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
Sep 24 20:17:20 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:17:21.002938 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.986279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.29.83.129:37675)
0924 20:17:20.986541 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:20.986551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:20.986579 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:17:20.986631 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:20.986634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:20.986803 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:20.986950 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:20.987601 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.988458 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.989662 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.990248 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.990414 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.992933 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:20.992943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:20.992956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:20.992984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:21.000715 (+ 7731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.001352 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.001357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.001359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.001445 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.001788 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.001792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.001794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.002062 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:21.002229 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.002582 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.002750 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260924 20:17:21.002938 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:20.985502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.0.0.1:37782)
0924 20:17:20.985889 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:20.985931 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:20.986051 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:20.986953 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:20.986960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:20.987369 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:20.987476 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.987483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.988573 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.988578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.989513 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:20.989525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:20.990422 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:20.990426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:20.990539 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:20.991197 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:20.991218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:20.992793 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:21.000882 (+ 8089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.000897 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.000904 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.001201 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.001554 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.001558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.001560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.001657 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.002193 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:21.002200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:21.002346 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.002499 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.002751 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":109,"threads_started":1}
I20260924 20:17:21.004195 11382 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de002dee89e472b86708741a9243b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:20 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37782
I20260924 20:17:21.005100 11457 tablet_replica.cc:330] stopping tablet replica
I20260924 20:17:21.005424 11457 raft_consensus.cc:2229] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:21.005692 11457 raft_consensus.cc:2258] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:21.006821 11457 ts_tablet_manager.cc:1905] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:21.010607 11457 ts_tablet_manager.cc:1918] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:21.010696 11457 log.cc:1196] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/6de002dee89e472b86708741a9243b2f
I20260924 20:17:21.011042 11457 ts_tablet_manager.cc:1939] T 6de002dee89e472b86708741a9243b2f P 076c59b72cbc438daaca427a3f570f64: Deleting consensus metadata
I20260924 20:17:21.012990 11246 catalog_manager.cc:4841] TS 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675): tablet 6de002dee89e472b86708741a9243b2f (table test-table [id=d25ad7aa70fa4fe8ade71db9a12ef2e5]) successfully deleted
W20260924 20:17:21.084867 11452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:21.085165 11452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:21.085250 11452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:21.089573 11452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:21.089668 11452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:21.089807 11452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:21.095393 11452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=40665
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:21.096540 11452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:21.097580 11452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:21.107764 11462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:21.107968 11465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.107982 11452 server_base.cc:1034] running on GCE node
W20260924 20:17:21.107776 11463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.108582 11452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:21.109290 11452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:21.110457 11452 hybrid_clock.cc:648] HybridClock initialized: now 1790281041110426 us; error 48 us; skew 500 ppm
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:21.113314 11452 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:21.114210 11452 webserver.cc:460] Webserver started at http://127.29.83.130:40665/ using document root <none> and password file <none>
I20260924 20:17:21.114727 11452 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:21.114795 11452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:21.118590 11452 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:21.120692 11472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.122025 11452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:17:21.122151 11452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "c82210f4e0a7489fac84d5034c5b3d88"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "b53dcbfd58728bbfff56bbcbe4ff4a8f"
server_key_iv: "484d7c46cf4adeeedbee5782dddba73a"
server_key_version: "encryptionkey@0"
I20260924 20:17:21.122706 11452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:21.138890 11452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:21.142328 11452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:21.142539 11452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:21.143198 11452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:21.144083 11452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:21.144129 11452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.144171 11452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:21.144229 11452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.153451 11452 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:35363
I20260924 20:17:21.153475 11585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:35363 every 8 connection(s)
I20260924 20:17:21.154517 11452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:21.160203 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11452
I20260924 20:17:21.160629 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:42607
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=46611
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:21.168741 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.156387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:33289 (local address 127.29.83.190:38959)
0924 20:17:21.156558 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.156562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.157384 (+ 822us) server_negotiation.cc:407] Connection header received
0924 20:17:21.158846 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.158849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.158910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.158990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:21.160191 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.160873 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.162096 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.162344 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.162466 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.164975 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:21.164978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:21.164980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:21.165007 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:21.166597 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.167137 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.167141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.167147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.167225 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.167575 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.167580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.167582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.167771 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:21.167885 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.168421 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.168565 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:17:21.169574 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.156794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:33289)
0924 20:17:21.157238 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.157282 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.158623 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.159169 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.159176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.159541 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:21.160037 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.160050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.161011 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.161015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.161969 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.161979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.162510 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.162513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.162689 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.163328 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:21.163344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:21.164787 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:21.166723 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.166729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.166745 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.167018 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.167323 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.167326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.167328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.167479 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.167885 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:21.167893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:21.168155 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.168375 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.168713 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260924 20:17:21.170840 11586 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:21.171103 11586 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:21.171610 11586 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:21.172694 11260 ts_manager.cc:194] Registered new tserver with Master: c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
I20260924 20:17:21.173437 11260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:33289
W20260924 20:17:21.273572 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:21.273871 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:21.273916 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:21.278151 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:21.278213 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:21.278272 11589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:21.284000 11589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:42607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=46611
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:21.285089 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:21.286168 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:21.295748 11595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:21.295769 11594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:21.295740 11597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.296217 11589 server_base.cc:1034] running on GCE node
I20260924 20:17:21.296687 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:21.297324 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:21.298491 11589 hybrid_clock.cc:648] HybridClock initialized: now 1790281041298464 us; error 36 us; skew 500 ppm
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:21.301311 11589 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:21.302249 11589 webserver.cc:460] Webserver started at http://127.29.83.131:46611/ using document root <none> and password file <none>
I20260924 20:17:21.302832 11589 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:21.302958 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:21.307202 11589 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:17:21.309495 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.311025 11589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:21.311172 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "01da3a139bbc409aad6a9846818265bf"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "477ead512acc5b4cd9496d314b69ddb4"
server_key_iv: "c67e2198821208dd62a06156f966487d"
server_key_version: "encryptionkey@0"
I20260924 20:17:21.311785 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:21.322974 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:21.326340 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:21.326576 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:21.327234 11589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:21.328053 11589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:21.328100 11589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.328140 11589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:21.328197 11589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.340728 11589 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42607
I20260924 20:17:21.340762 11717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42607 every 8 connection(s)
I20260924 20:17:21.342294 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:21.347124 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11589
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:21.358121 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.347894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.29.83.190:38959)
0924 20:17:21.348614 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.348618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.348628 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:17:21.348654 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.348656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.348715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.349732 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:21.349868 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.350643 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.351661 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.352074 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.352194 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.354083 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:21.354087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:21.354090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:21.354115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:21.356084 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.356578 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.356581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.356583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.356675 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.357013 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.357019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.357022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.357279 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:21.357521 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.357798 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.357934 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"spinlock_wait_cycles":26624,"thread_start_us":93,"threads_started":1}
I20260924 20:17:21.361189 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.346627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:54977 (local address 127.29.83.190:38959)
0924 20:17:21.346868 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.346873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.346892 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:17:21.348645 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.348648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.348715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.348784 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:21.350697 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.351352 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.353098 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.353450 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.353570 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.356593 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:21.356596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:21.356599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:21.356622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:21.358744 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.359358 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.359361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.359363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.359448 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:21.359857 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:21.359862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:21.359864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:21.360101 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:21.360212 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.360888 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.361029 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"spinlock_wait_cycles":9088}
I20260924 20:17:21.362308 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.345297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:54977)
0924 20:17:21.345925 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.345980 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.348392 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.349262 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.349272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.349819 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:21.350478 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.350497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.351498 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.351505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.352944 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.352958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.353636 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.353643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.353919 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.354582 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:21.354605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:21.356386 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:21.358863 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.358869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.358887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.359217 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.359554 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:21.359557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:21.359558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:21.359729 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:21.360220 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:21.360225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:21.360600 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.360811 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.361242 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":173,"threads_started":1}
I20260924 20:17:21.363523 11718 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38959
I20260924 20:17:21.363829 11718 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:21.364547 11718 heartbeater.cc:511] Master 127.29.83.190:38959 requested a full tablet report, sending...
I20260924 20:17:21.365653 11260 ts_manager.cc:194] Registered new tserver with Master: 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607)
I20260924 20:17:21.366245 11260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:54977
I20260924 20:17:21.372365 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:17:21.376777 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.372936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.29.83.190:38959)
0924 20:17:21.373081 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.373086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.373237 (+ 151us) server_negotiation.cc:407] Connection header received
0924 20:17:21.373394 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.373396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.373495 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.373581 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:17:21.373847 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.374362 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.375355 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.375569 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.375649 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.375936 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:17:21.376362 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:17:21.376454 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.376552 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.376602 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:17:21.377473 11260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:17:21.380851 11260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:17:21.388898 11382 tablet_service.cc:1467] Processing CreateTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad (DEFAULT_TABLE table=test-table [id=f1d1d671d0a247088f541475cc4dfc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:21.389587 11382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d2d7e3b6f48c5b2bf5e3387170aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:21.392335 11340 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Bootstrap starting.
I20260924 20:17:21.393235 11340 tablet_bootstrap.cc:654] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:21.394317 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.388141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.0.0.1:59404)
0924 20:17:21.388393 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.388407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.388497 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.389284 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.389288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.389310 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:21.389405 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.389411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.391189 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.391192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.392391 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.392402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.393799 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.393805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.393913 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.393968 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.394068 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.394152 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":28,"thread_start_us":85,"threads_started":1}
I20260924 20:17:21.394443 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.388551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.29.83.130:35363)
0924 20:17:21.388871 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.388883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.388903 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:17:21.388970 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.388974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.389145 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.389296 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:21.389508 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.391038 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.392850 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.393631 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.393724 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.393908 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.394127 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.394240 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
I20260924 20:17:21.395176 11340 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: No bootstrap required, opened a new log
I20260924 20:17:21.395272 11340 ts_tablet_manager.cc:1397] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:21.395510 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.387891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42607 (local address 127.0.0.1:41556)
0924 20:17:21.388019 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.388035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.388115 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.389191 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.389197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.389220 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:21.389334 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.389344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.391242 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.391246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.392090 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.392100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.395120 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.395124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.395230 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.395254 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.395327 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.395415 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":37}
I20260924 20:17:21.395720 11340 raft_consensus.cc:348] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.395866 11340 raft_consensus.cc:374] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:21.395908 11340 raft_consensus.cc:729] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076c59b72cbc438daaca427a3f570f64, State: Initialized, Role: FOLLOWER
I20260924 20:17:21.396044 11340 consensus_queue.cc:260] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.396564 11340 ts_tablet_manager.cc:1428] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:17:21.398159 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.388043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.29.83.131:42607)
0924 20:17:21.388626 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.388636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.388655 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:17:21.388752 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.388758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.389017 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.389653 (+ 636us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:21.389850 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.391064 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.394126 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.394984 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.395103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.397842 (+ 2739us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.397904 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.398014 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":135,"threads_started":1}
I20260924 20:17:21.397650 11520 tablet_service.cc:1467] Processing CreateTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad (DEFAULT_TABLE table=test-table [id=f1d1d671d0a247088f541475cc4dfc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:21.398875 11520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d2d7e3b6f48c5b2bf5e3387170aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:21.400846 11652 tablet_service.cc:1467] Processing CreateTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad (DEFAULT_TABLE table=test-table [id=f1d1d671d0a247088f541475cc4dfc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:21.401857 11652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812d2d7e3b6f48c5b2bf5e3387170aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:21.406163 11731 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Bootstrap starting.
I20260924 20:17:21.408713 11732 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Bootstrap starting.
I20260924 20:17:21.409044 11731 tablet_bootstrap.cc:654] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:21.409973 11731 log.cc:824] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:21.410830 11732 tablet_bootstrap.cc:654] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:21.412103 11732 log.cc:824] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:21.412563 11731 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: No bootstrap required, opened a new log
I20260924 20:17:21.412784 11731 ts_tablet_manager.cc:1397] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:17:21.414309 11732 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: No bootstrap required, opened a new log
I20260924 20:17:21.414484 11732 ts_tablet_manager.cc:1397] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:17:21.416754 11731 raft_consensus.cc:348] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.417029 11731 raft_consensus.cc:374] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:21.417081 11731 raft_consensus.cc:729] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82210f4e0a7489fac84d5034c5b3d88, State: Initialized, Role: FOLLOWER
I20260924 20:17:21.417557 11731 consensus_queue.cc:260] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.418282 11586 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
I20260924 20:17:21.418570 11731 ts_tablet_manager.cc:1428] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:17:21.418967 11732 raft_consensus.cc:348] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.419217 11732 raft_consensus.cc:374] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:21.419286 11732 raft_consensus.cc:729] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01da3a139bbc409aad6a9846818265bf, State: Initialized, Role: FOLLOWER
I20260924 20:17:21.419663 11732 consensus_queue.cc:260] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.420307 11718 heartbeater.cc:503] Master 127.29.83.190:38959 was elected leader, sending a full tablet report...
I20260924 20:17:21.420557 11732 ts_tablet_manager.cc:1428] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 20:17:21.459875 11449 tablet.cc:2369] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:17:21.469000 11735 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:21.469280 11735 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.470541 11735 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675)
I20260924 20:17:21.474545 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.470742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42607 (local address 127.29.83.130:48203)
0924 20:17:21.470874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.470891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.470982 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.471359 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.471364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.471426 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:21.471562 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.471569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.472446 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.472450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.473286 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.473295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.474088 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.474092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.474199 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.474225 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.474305 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.474396 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":8192}
I20260924 20:17:21.474575 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.470955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:48203 (local address 127.29.83.131:42607)
0924 20:17:21.471089 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.471095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.471109 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:17:21.471162 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.471165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.471212 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.471296 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:21.471713 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.472333 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.473427 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.473967 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.474061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.474100 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.474341 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.474430 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:17:21.474779 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.470831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.29.83.130:57309)
0924 20:17:21.471276 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.471290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:21.471423 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:21.471738 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:21.471740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:21.471759 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:21.471863 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.471869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.472698 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.472701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.473541 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:21.473551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.474353 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.474356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:21.474454 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.474477 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:17:21.474559 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:17:21.474633 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"spinlock_wait_cycles":37888,"thread_start_us":146,"threads_started":1}
I20260924 20:17:21.474787 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.470948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:57309 (local address 127.29.83.129:37675)
0924 20:17:21.471077 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.471081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.471375 (+ 294us) server_negotiation.cc:407] Connection header received
0924 20:17:21.471538 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.471542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.471610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.471721 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:21.471975 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.472550 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.473670 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.474218 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.474343 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.474498 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.474576 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.474653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:17:21.475255 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812d2d7e3b6f48c5b2bf5e3387170aad" candidate_uuid: "c82210f4e0a7489fac84d5034c5b3d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01da3a139bbc409aad6a9846818265bf" is_pre_election: true
I20260924 20:17:21.475649 11660 raft_consensus.cc:2454] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82210f4e0a7489fac84d5034c5b3d88 in term 0.
I20260924 20:17:21.475747 11402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812d2d7e3b6f48c5b2bf5e3387170aad" candidate_uuid: "c82210f4e0a7489fac84d5034c5b3d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "076c59b72cbc438daaca427a3f570f64" is_pre_election: true
I20260924 20:17:21.476063 11402 raft_consensus.cc:2454] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82210f4e0a7489fac84d5034c5b3d88 in term 0.
I20260924 20:17:21.476199 11473 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88; no voters:
I20260924 20:17:21.476653 11735 raft_consensus.cc:2781] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:21.476747 11735 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:21.476799 11735 raft_consensus.cc:3037] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:21.478305 11735 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.478762 11735 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [CANDIDATE]: Term 1 election: Requested vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675)
I20260924 20:17:21.479065 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812d2d7e3b6f48c5b2bf5e3387170aad" candidate_uuid: "c82210f4e0a7489fac84d5034c5b3d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01da3a139bbc409aad6a9846818265bf"
I20260924 20:17:21.479204 11660 raft_consensus.cc:3037] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:21.479220 11402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812d2d7e3b6f48c5b2bf5e3387170aad" candidate_uuid: "c82210f4e0a7489fac84d5034c5b3d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "076c59b72cbc438daaca427a3f570f64"
I20260924 20:17:21.479370 11402 raft_consensus.cc:3037] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:21.480329 11660 raft_consensus.cc:2454] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82210f4e0a7489fac84d5034c5b3d88 in term 1.
I20260924 20:17:21.480392 11402 raft_consensus.cc:2454] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82210f4e0a7489fac84d5034c5b3d88 in term 1.
I20260924 20:17:21.480602 11473 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88; no voters:
I20260924 20:17:21.480756 11735 raft_consensus.cc:2781] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:21.481005 11735 raft_consensus.cc:686] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 1 LEADER]: Becoming Leader. State: Replica: c82210f4e0a7489fac84d5034c5b3d88, State: Running, Role: LEADER
I20260924 20:17:21.481403 11735 consensus_queue.cc:237] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:21.484566 11259 catalog_manager.cc:5495] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 reported cstate change: term changed from 0 to 1, leader changed from <none> to c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "c82210f4e0a7489fac84d5034c5b3d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } health_report { overall_health: UNKNOWN } } }
I20260924 20:17:21.518889 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.515361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.29.83.130:35363)
0924 20:17:21.515491 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:21.515495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:21.515510 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:17:21.515637 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:21.515640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:21.515754 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:21.515848 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:17:21.516130 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.516628 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.517418 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:21.517622 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:21.517696 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:21.518058 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:17:21.518488 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:17:21.518587 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:21.518636 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 20:17:21.518693 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:17:21.524971 11660 raft_consensus.cc:1261] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Refusing update from remote peer c82210f4e0a7489fac84d5034c5b3d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:21.524968 11402 raft_consensus.cc:1261] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Refusing update from remote peer c82210f4e0a7489fac84d5034c5b3d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:21.525692 11738 consensus_queue.cc:1035] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:21.525938 11735 consensus_queue.cc:1035] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:21.542220 11259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:21.542407 11259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:21.544850 11259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Sending DeleteTablet for 3 replicas of tablet 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:21.545583 11382 tablet_service.cc:1514] Processing DeleteTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:21 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:37782
I20260924 20:17:21.545608 11520 tablet_service.cc:1514] Processing DeleteTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:21 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:59404
I20260924 20:17:21.546139 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11298
I20260924 20:17:21.546129 11751 tablet_replica.cc:330] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: stopping tablet replica
I20260924 20:17:21.546303 11652 tablet_service.cc:1514] Processing DeleteTablet for tablet 812d2d7e3b6f48c5b2bf5e3387170aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:21 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:41556
I20260924 20:17:21.546828 11753 tablet_replica.cc:330] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: stopping tablet replica
I20260924 20:17:21.547137 11753 raft_consensus.cc:2229] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:21.547430 11753 raft_consensus.cc:2258] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:21.549098 11753 ts_tablet_manager.cc:1905] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:21.549472 11752 tablet_replica.cc:330] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: stopping tablet replica
I20260924 20:17:21.549849 11752 raft_consensus.cc:2229] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:17:21.550349 11752 raft_consensus.cc:2258] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:21.551913 11752 ts_tablet_manager.cc:1905] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:21.552752 11753 ts_tablet_manager.cc:1918] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:21.552881 11753 log.cc:1196] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:21.553247 11753 ts_tablet_manager.cc:1939] T 812d2d7e3b6f48c5b2bf5e3387170aad P 01da3a139bbc409aad6a9846818265bf: Deleting consensus metadata
W20260924 20:17:21.554047 11246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:37675 (error 108)
I20260924 20:17:21.554628 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11452
I20260924 20:17:21.555846 11244 catalog_manager.cc:4841] TS 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): tablet 812d2d7e3b6f48c5b2bf5e3387170aad (table test-table [id=f1d1d671d0a247088f541475cc4dfc76]) successfully deleted
I20260924 20:17:21.556526 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.556151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37675 (local address 127.0.0.1:37798)
0924 20:17:21.556339 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.556425 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 20:17:21.556713 11246 catalog_manager.cc:4584] TS 076c59b72cbc438daaca427a3f570f64 (127.29.83.129:37675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 812d2d7e3b6f48c5b2bf5e3387170aad: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37675: connect: Connection refused (error 111)
I20260924 20:17:21.561443 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11589
I20260924 20:17:21.561707 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:21.561379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.0.0.1:59430)
0924 20:17:21.561531 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:21.561599 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.130:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 20:17:21.567446 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11227
I20260924 20:17:21.574219 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--webserver_interface=127.29.83.190
--webserver_port=43449
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:21.683990 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:21.684331 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:21.684419 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:21.688476 11755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:21.688584 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:21.688625 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:21.688663 11755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:21.688694 11755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:21.694240 11755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:21.695861 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:21.697062 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:21.706197 11761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:21.706197 11763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.706511 11755 server_base.cc:1034] running on GCE node
W20260924 20:17:21.706189 11760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.707227 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:21.708392 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:21.709591 11755 hybrid_clock.cc:648] HybridClock initialized: now 1790281041709578 us; error 28 us; skew 500 ppm
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:21.712677 11755 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:21.713627 11755 webserver.cc:460] Webserver started at http://127.29.83.190:43449/ using document root <none> and password file <none>
I20260924 20:17:21.714262 11755 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:21.714356 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:21.718205 11755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:17:21.720412 11770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.721657 11755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:17:21.721848 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "45b051c9dfea47b1a747cc7da7407d85"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "b55db557e82db4234b67bc5831686676"
server_key_iv: "35a8e48fedad6a6f5b49b7333ded1507"
server_key_version: "encryptionkey@0"
I20260924 20:17:21.722420 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:21.739125 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:21.742301 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:21.742525 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:21.750080 11755 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38959
I20260924 20:17:21.750387 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38959 every 8 connection(s)
I20260924 20:17:21.751188 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:21.755545 11823 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:17:21.757287 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap starting.
I20260924 20:17:21.760883 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11755
I20260924 20:17:21.761312 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:37675
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=34355
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:21.775592 11823 log.cc:824] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:21.787143 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:21.787544 11823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85: Bootstrap complete.
I20260924 20:17:21.791692 11823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:21.792073 11823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Initialized, Role: FOLLOWER
I20260924 20:17:21.792570 11823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:21.792689 11823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:21.792785 11823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:21.792888 11823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:17:21.794927 11823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:21.795284 11823 leader_election.cc:302] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b051c9dfea47b1a747cc7da7407d85; no voters:
I20260924 20:17:21.795646 11823 leader_election.cc:288] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:17:21.795740 11827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:17:21.795936 11827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [term 3 LEADER]: Becoming Leader. State: Replica: 45b051c9dfea47b1a747cc7da7407d85, State: Running, Role: LEADER
I20260924 20:17:21.796231 11827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } }
I20260924 20:17:21.796890 11823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:21.797206 11827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b051c9dfea47b1a747cc7da7407d85. Latest consensus state: current_term: 3 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:21.797329 11827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:21.797705 11831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:21.797850 11828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "45b051c9dfea47b1a747cc7da7407d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b051c9dfea47b1a747cc7da7407d85" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38959 } } }
I20260924 20:17:21.798027 11828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b051c9dfea47b1a747cc7da7407d85 [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:21.803025 11831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:21.803730 11831 catalog_manager.cc:1259] Loaded cluster ID: b3848e7c62694eee9f5d4c041e65b56c
I20260924 20:17:21.803807 11831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 20:17:21.804589 11831 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7cc54e19ad google::LogMessage::Fail() at ??:0
@ 0x7f7cc54e488a google::LogMessage::SendToLog() at ??:0
@ 0x7f7cc54e14de google::LogMessage::Flush() at ??:0
@ 0x7f7cc54e2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ccfe24fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7ccfe25d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7ccfe22579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7ccfe66640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7cc77333d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7cc7733c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7cc77353f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7cc7724512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7cc8d686db start_thread at ??:0
@ 0x7f7cc3fdf61f clone at ??:0
W20260924 20:17:21.911128 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:21.911550 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:21.911619 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:21.918009 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:21.918107 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:21.918198 11825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:21.926415 11825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:37675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=34355
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:21.927649 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:21.928733 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:21.939181 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:21.939167 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.939461 11825 server_base.cc:1034] running on GCE node
W20260924 20:17:21.939759 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:21.940348 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:21.941329 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:21.942560 11825 hybrid_clock.cc:648] HybridClock initialized: now 1790281041942535 us; error 32 us; skew 500 ppm
Sep 24 20:17:21 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:21.946288 11825 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:21.947479 11825 webserver.cc:460] Webserver started at http://127.29.83.129:34355/ using document root <none> and password file <none>
I20260924 20:17:21.948180 11825 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:21.948278 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:21.953120 11825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:21.955927 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:21.957671 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 20:17:21.957846 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "076c59b72cbc438daaca427a3f570f64"
format_stamp: "Formatted at 2026-09-24 20:17:19 on dist-test-slave-52fr"
server_key: "2f8948c727a54cdd4b28c66f8672fc74"
server_key_iv: "fd7d6e36537a494d43442238e2489207"
server_key_version: "encryptionkey@0"
I20260924 20:17:21.958703 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:21.979671 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:21.984064 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:21.984279 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:21.985085 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:21.986521 11881 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:17:21.990736 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:17:21.990820 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260924 20:17:21.990891 11825 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:17:21.993114 11825 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:17:21.993175 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:17:21.993521 11881 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Bootstrap starting.
I20260924 20:17:22.004343 11825 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:37675
I20260924 20:17:22.004558 11988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:37675 every 8 connection(s)
I20260924 20:17:22.005663 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:22.009363 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11825
I20260924 20:17:22.009948 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:35363
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=40665
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:22.013909 11881 log.cc:824] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:22.024035 11881 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:22.024446 11881 tablet_bootstrap.cc:492] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Bootstrap complete.
I20260924 20:17:22.025884 11881 ts_tablet_manager.cc:1397] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20260924 20:17:22.029081 11881 raft_consensus.cc:348] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:22.029624 11881 raft_consensus.cc:729] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076c59b72cbc438daaca427a3f570f64, State: Initialized, Role: FOLLOWER
I20260924 20:17:22.030208 11881 consensus_queue.cc:260] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:22.031348 11881 ts_tablet_manager.cc:1428] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260924 20:17:22.070230 11991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:17:22.008032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:41085)
0924 20:17:22.008633 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.008695 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:22.009751 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:22.069324 (+ 59573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":164,"threads_started":1}
W20260924 20:17:22.070735 11989 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:38959 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:38959: Client connection negotiation failed: client connection to 127.29.83.190:38959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:17:22.071472 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.071193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:37363)
0924 20:17:22.071343 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.071416 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 20:17:22.072081 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.071883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:36351)
0924 20:17:22.071985 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.072037 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 20:17:22.072233 11989 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:17:22.133796 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:22.134140 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:22.134227 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:22.138545 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:22.138655 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:22.138758 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:22.144512 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=40665
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:22.145696 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:22.146968 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:22.156917 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:22.156986 11992 server_base.cc:1034] running on GCE node
W20260924 20:17:22.156898 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:22.156903 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:22.157598 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:22.158305 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:22.159529 11992 hybrid_clock.cc:648] HybridClock initialized: now 1790281042159509 us; error 30 us; skew 500 ppm
Sep 24 20:17:22 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281042, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:22.162544 11992 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:22.163507 11992 webserver.cc:460] Webserver started at http://127.29.83.130:40665/ using document root <none> and password file <none>
I20260924 20:17:22.164095 11992 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:22.164191 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:22.168399 11992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:22.170725 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:22.171988 11992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:17:22.172125 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "c82210f4e0a7489fac84d5034c5b3d88"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "b53dcbfd58728bbfff56bbcbe4ff4a8f"
server_key_iv: "484d7c46cf4adeeedbee5782dddba73a"
server_key_version: "encryptionkey@0"
I20260924 20:17:22.172693 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:22.198118 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:22.201153 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:22.201364 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:22.202059 11992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:22.203380 12016 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 20:17:22.207569 12016 ts_tablet_manager.cc:1852] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 20:17:22.208022 12016 ts_tablet_manager.cc:1905] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:22.211989 12016 ts_tablet_manager.cc:1918] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:22.212105 12016 log.cc:1196] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:22.212457 12016 ts_tablet_manager.cc:1939] T 812d2d7e3b6f48c5b2bf5e3387170aad P c82210f4e0a7489fac84d5034c5b3d88: Deleting consensus metadata
I20260924 20:17:22.212793 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 20:17:22.212888 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20260924 20:17:22.212985 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:22.213035 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:22.222651 11992 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:35363
I20260924 20:17:22.222687 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:35363 every 8 connection(s)
I20260924 20:17:22.223533 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:22.226958 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.225650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:46125)
0924 20:17:22.226136 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.226250 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":141,"threads_started":1}
W20260924 20:17:22.227415 12124 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:38959 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:38959: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
I20260924 20:17:22.228039 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.227796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:52675)
0924 20:17:22.227925 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.227979 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:17:22.228708 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.228508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:47619)
0924 20:17:22.228615 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.228659 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:17:22.228893 12124 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:17:22.229117 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 11992
I20260924 20:17:22.229516 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:42607
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:38959
--webserver_port=46611
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:22.339109 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:22.339380 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:22.339424 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:22.343458 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:22.343528 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:22.343590 12127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:22.349061 12127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:42607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=46611
--tserver_master_addrs=127.29.83.190:38959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:22.350224 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:22.351207 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:22.361289 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:22.361294 12127 server_base.cc:1034] running on GCE node
W20260924 20:17:22.361313 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:22.361326 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:22.362139 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:22.362828 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:22.364054 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790281042364049 us; error 48 us; skew 500 ppm
Sep 24 20:17:22 dist-test-slave-52fr krb5kdc[10677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281042, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:22.366997 12127 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:22.367946 12127 webserver.cc:460] Webserver started at http://127.29.83.131:46611/ using document root <none> and password file <none>
I20260924 20:17:22.368542 12127 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:22.368673 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:22.372862 12127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:22.375150 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:22.376525 12127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:17:22.376672 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "01da3a139bbc409aad6a9846818265bf"
format_stamp: "Formatted at 2026-09-24 20:17:20 on dist-test-slave-52fr"
server_key: "477ead512acc5b4cd9496d314b69ddb4"
server_key_iv: "c67e2198821208dd62a06156f966487d"
server_key_version: "encryptionkey@0"
I20260924 20:17:22.377374 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:22.392927 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:22.396066 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:22.396297 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:22.396927 12127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:22.397802 12127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:22.397881 12127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:22.397958 12127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:22.398002 12127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:22.407469 12127 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:42607
I20260924 20:17:22.407500 12255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:42607 every 8 connection(s)
I20260924 20:17:22.408547 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:22.412519 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.410781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:35769)
0924 20:17:22.411257 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.411376 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
W20260924 20:17:22.413035 12256 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:38959 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:38959: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
I20260924 20:17:22.413628 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.413416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:42185)
0924 20:17:22.413526 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.413575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 20:17:22.414248 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:22.414063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:39047)
0924 20:17:22.414158 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:22.414200 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 20:17:22.414438 12256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:17:22.416024 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12127
I20260924 20:17:23.073392 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.072903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:48613)
0924 20:17:23.073199 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:23.073265 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260924 20:17:23.230229 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.229672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:47547)
0924 20:17:23.230027 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:23.230116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
I20260924 20:17:23.415480 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.415056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:58765)
0924 20:17:23.415313 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:23.415376 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260924 20:17:23.901611 12263 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:23.901914 12263 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:23.903131 12263 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
Sep 24 20:17:23 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.130@KRBTEST.COM
Sep 24 20:17:23 dist-test-slave-52fr krb5kdc[10677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281041, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.131@KRBTEST.COM
I20260924 20:17:23.917927 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.903547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:42069 (local address 127.29.83.131:42607)
0924 20:17:23.904034 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:23.904056 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:23.904101 (+ 45us) server_negotiation.cc:407] Connection header received
0924 20:17:23.904769 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:23.904775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:23.905121 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:23.905437 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:23.906825 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.907895 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909115 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.909560 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909730 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:23.912946 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:23.912950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:23.912962 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:23.912990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:23.915084 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:23.915830 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:23.915837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:23.915845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:23.915996 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:23.916377 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:23.916380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:23.916382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:23.916774 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:23.916920 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:23.917459 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 20:17:23.917718 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":121,"threads_started":1}
I20260924 20:17:23.917995 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.904091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34937 (local address 127.29.83.130:35363)
0924 20:17:23.904386 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:23.904396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:23.904427 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:17:23.905091 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:23.905098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:23.905420 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:23.905703 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:23.906792 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.907877 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909128 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.909562 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909734 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:23.912916 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:23.912921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:23.912933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:23.912961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:23.915344 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:23.915866 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:23.915871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:23.915879 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:23.915997 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:23.916360 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:23.916365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:23.916367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:23.916774 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:23.916920 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:23.917460 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 20:17:23.917740 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260924 20:17:23.918100 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.903702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35363 (local address 127.29.83.129:34937)
0924 20:17:23.904015 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:23.904029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:23.904122 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:23.905642 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:23.905645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:23.906079 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:23.906613 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:23.906630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.908025 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.908030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:23.908999 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:23.909009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.909689 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:23.909937 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:23.910596 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:23.910612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:23.912258 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:23.915452 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:23.915457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:23.915460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:23.915756 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:23.916101 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:23.916106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:23.916108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:23.916256 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:23.916902 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:23.916905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:23.917153 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 20:17:23.917347 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 20:17:23.917824 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":262,"spinlock_wait_cycles":5120,"thread_start_us":58,"threads_started":1}
I20260924 20:17:23.918159 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:23.903324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:42607 (local address 127.29.83.129:42069)
0924 20:17:23.903624 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:23.903640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:23.903750 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:23.905497 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:23.905503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:23.906068 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:23.906613 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:23.906629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.908026 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.908029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:23.908990 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:23.908999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:23.909681 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:23.909684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:23.909932 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:23.910652 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:23.910664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:23.912644 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:23.915220 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:23.915226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:23.915240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:23.915682 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:23.916110 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:23.916115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:23.916117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:23.916265 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:23.916897 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:23.916903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:23.917152 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 20:17:23.917348 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 20:17:23.917705 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":259,"spinlock_wait_cycles":18560,"thread_start_us":93,"threads_started":1}
W20260924 20:17:23.920903 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
W20260924 20:17:23.921187 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:23.921276 11875 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64; no voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:23.921583 12263 raft_consensus.cc:2726] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:17:24.074605 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:24.074295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:34893)
0924 20:17:24.074454 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:24.074519 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 20:17:24.231652 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:24.231001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:34069)
0924 20:17:24.231431 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:24.231510 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":75,"threads_started":1}
I20260924 20:17:24.416859 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:24.416303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:47587)
0924 20:17:24.416598 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:24.416730 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260924 20:17:25.075855 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:25.075376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:41593)
0924 20:17:25.075689 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:25.075757 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260924 20:17:25.232969 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:25.232462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:34903)
0924 20:17:25.232784 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:25.232847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
I20260924 20:17:25.418329 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:25.417904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:33767)
0924 20:17:25.418170 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:25.418235 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260924 20:17:26.068059 12273 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:26.068185 12273 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:26.068742 12273 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
W20260924 20:17:26.069664 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
W20260924 20:17:26.069862 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:26.069947 11875 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64; no voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:26.070112 12273 raft_consensus.cc:2726] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:17:26.077315 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:26.076714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:56181)
0924 20:17:26.077055 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:26.077156 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
I20260924 20:17:26.234316 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:26.233858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:47423)
0924 20:17:26.234132 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:26.234202 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260924 20:17:26.419538 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:26.419106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:37475)
0924 20:17:26.419390 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:26.419448 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260924 20:17:27.078814 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:27.078333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:53945)
0924 20:17:27.078644 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:27.078708 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260924 20:17:27.235733 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:27.235234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:60463)
0924 20:17:27.235540 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:27.235606 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260924 20:17:27.420846 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:27.420353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:43213)
0924 20:17:27.420682 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:27.420747 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260924 20:17:27.971490 12280 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:27.971622 12280 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:27.971979 12280 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
W20260924 20:17:27.972858 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
W20260924 20:17:27.972970 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:27.973045 11875 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64; no voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:27.973243 12280 raft_consensus.cc:2726] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:17:28.080303 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:28.079814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:60653)
0924 20:17:28.080099 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:28.080159 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260924 20:17:28.237270 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:28.236794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:44747)
0924 20:17:28.237081 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:28.237168 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
I20260924 20:17:28.422225 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:28.421710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:41025)
0924 20:17:28.422073 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:28.422133 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":163,"threads_started":1}
I20260924 20:17:29.081564 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:29.081155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:36383)
0924 20:17:29.081407 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:29.081468 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260924 20:17:29.238607 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:29.238125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:32927)
0924 20:17:29.238445 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:29.238509 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":75,"threads_started":1}
I20260924 20:17:29.423584 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:29.423053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:47337)
0924 20:17:29.423406 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:29.423488 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
I20260924 20:17:30.082947 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:30.082478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:54499)
0924 20:17:30.082791 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:30.082849 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
I20260924 20:17:30.239917 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:30.239455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:37875)
0924 20:17:30.239754 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:30.239814 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260924 20:17:30.425101 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:30.424622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:60163)
0924 20:17:30.424931 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:30.424998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260924 20:17:31.064459 12290 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:31.064602 12290 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:31.065021 12290 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
W20260924 20:17:31.066092 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
W20260924 20:17:31.066236 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:31.066294 11875 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64; no voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:31.066447 12290 raft_consensus.cc:2726] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:17:31.084161 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:31.083750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:47801)
0924 20:17:31.083995 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:31.084064 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260924 20:17:31.241264 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:31.240776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:58853)
0924 20:17:31.241087 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:31.241158 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260924 20:17:31.426504 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:31.426051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:52141)
0924 20:17:31.426344 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:31.426409 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
I20260924 20:17:32.085536 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:32.085098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:33321)
0924 20:17:32.085374 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:32.085434 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260924 20:17:32.242748 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:32.242311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:41387)
0924 20:17:32.242580 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:32.242647 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260924 20:17:32.427680 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:32.427278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:58541)
0924 20:17:32.427526 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:32.427584 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260924 20:17:33.086897 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:33.086403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:44259)
0924 20:17:33.086730 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:33.086796 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260924 20:17:33.244153 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:33.243648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:54337)
0924 20:17:33.243970 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:33.244041 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260924 20:17:33.428951 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:33.428465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:55369)
0924 20:17:33.428767 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:33.428834 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260924 20:17:34.088344 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:34.087901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:36911)
0924 20:17:34.088178 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:34.088234 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260924 20:17:34.245445 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:34.244986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:59845)
0924 20:17:34.245274 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:34.245336 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260924 20:17:34.430365 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:34.429894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:36031)
0924 20:17:34.430208 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:34.430265 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260924 20:17:35.089677 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:35.089221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:51845)
0924 20:17:35.089496 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:35.089578 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260924 20:17:35.246898 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:35.246441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:36315)
0924 20:17:35.246722 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:35.246799 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260924 20:17:35.432053 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:35.431499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:60457)
0924 20:17:35.431844 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:35.431913 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260924 20:17:36.091245 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:36.090714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:51223)
0924 20:17:36.091031 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:36.091094 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260924 20:17:36.248274 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:36.247816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:52495)
0924 20:17:36.248100 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:36.248166 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260924 20:17:36.433406 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:36.432906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.131:37837)
0924 20:17:36.433208 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:36.433263 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260924 20:17:37.061936 12309 raft_consensus.cc:482] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:37.062099 12309 raft_consensus.cc:504] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01da3a139bbc409aad6a9846818265bf" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 42607 } } peers { permanent_uuid: "c82210f4e0a7489fac84d5034c5b3d88" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35363 } } peers { permanent_uuid: "076c59b72cbc438daaca427a3f570f64" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37675 } }
I20260924 20:17:37.062526 12309 leader_election.cc:288] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607), c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363)
W20260924 20:17:37.063453 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01da3a139bbc409aad6a9846818265bf (127.29.83.131:42607): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
W20260924 20:17:37.063570 11875 leader_election.cc:341] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c82210f4e0a7489fac84d5034c5b3d88 (127.29.83.130:35363): Not found: Tablet not found: 812d2d7e3b6f48c5b2bf5e3387170aad
I20260924 20:17:37.063711 11875 leader_election.cc:302] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 076c59b72cbc438daaca427a3f570f64; no voters: 01da3a139bbc409aad6a9846818265bf, c82210f4e0a7489fac84d5034c5b3d88
I20260924 20:17:37.063899 12309 raft_consensus.cc:2726] T 812d2d7e3b6f48c5b2bf5e3387170aad P 076c59b72cbc438daaca427a3f570f64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:17:37.092921 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:37.092420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.129:57233)
0924 20:17:37.092746 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:37.092805 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260924 20:17:37.249897 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:37.249283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38959 (local address 127.29.83.130:51273)
0924 20:17:37.249689 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:37.249783 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":101,"threads_started":1}
I20260924 20:17:37.422358 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11825
I20260924 20:17:37.428189 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 11992
I20260924 20:17:37.433511 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12127
2026-09-24T20:17:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19257 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:17:37 dist-test-slave-52fr krb5kdc[12314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:17:37 dist-test-slave-52fr krb5kdc[12314](info): set up 2 sockets
Sep 24 20:17:37 dist-test-slave-52fr krb5kdc[12314](info): commencing operation
krb5kdc: starting...
W20260924 20:17:38.470821 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:17:38 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:17:38.597237 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:38.707870 12327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:38.708163 12327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:38.708204 12327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:38.712265 12327 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:17:38.712325 12327 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:17:38.712342 12327 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:17:38.712358 12327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:38.712376 12327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:38.712394 12327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:38.712410 12327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:38.712425 12327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:38.718335 12327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:38.719413 12327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:38.720395 12327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:38.729110 12335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:38.729143 12332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:38.729324 12327 server_base.cc:1034] running on GCE node
W20260924 20:17:38.729110 12333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:38.730078 12327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:38.731201 12327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:38.732404 12327 hybrid_clock.cc:648] HybridClock initialized: now 1790281058732376 us; error 46 us; skew 500 ppm
Sep 24 20:17:38 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281058, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:38.735376 12327 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:38.736335 12327 webserver.cc:460] Webserver started at http://127.29.83.190:34719/ using document root <none> and password file <none>
I20260924 20:17:38.736898 12327 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:38.736987 12327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:38.737228 12327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:38.739300 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "5e9bb2789d57410c9d1831638f6b569a"
format_stamp: "Formatted at 2026-09-24 20:17:38 on dist-test-slave-52fr"
server_key: "6aa52da0019bc7495d67e9493d5059ab"
server_key_iv: "54bdfeccf1253933fd092558e78a4a5c"
server_key_version: "encryptionkey@0"
I20260924 20:17:38.739851 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "5e9bb2789d57410c9d1831638f6b569a"
format_stamp: "Formatted at 2026-09-24 20:17:38 on dist-test-slave-52fr"
server_key: "6aa52da0019bc7495d67e9493d5059ab"
server_key_iv: "54bdfeccf1253933fd092558e78a4a5c"
server_key_version: "encryptionkey@0"
I20260924 20:17:38.743515 12327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:17:38.746052 12342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:38.747299 12327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:17:38.747439 12327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5e9bb2789d57410c9d1831638f6b569a"
format_stamp: "Formatted at 2026-09-24 20:17:38 on dist-test-slave-52fr"
server_key: "6aa52da0019bc7495d67e9493d5059ab"
server_key_iv: "54bdfeccf1253933fd092558e78a4a5c"
server_key_version: "encryptionkey@0"
I20260924 20:17:38.747550 12327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:38.782604 12327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:38.786201 12327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:38.786434 12327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:38.793558 12327 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36969
I20260924 20:17:38.793936 12394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36969 every 8 connection(s)
I20260924 20:17:38.794624 12327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:38.797374 12395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:38.803427 12395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap starting.
I20260924 20:17:38.804179 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12327
I20260924 20:17:38.804328 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:17:38.804616 30030 external_mini_cluster.cc:1428] Setting key 408f078a2bb1ed63774dc363177a7381
I20260924 20:17:38.806427 12395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:38.807211 12395 log.cc:824] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:38.809635 12395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: No bootstrap required, opened a new log
Sep 24 20:17:38 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:38.813963 12395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:38.814244 12395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:38.814307 12395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Initialized, Role: FOLLOWER
I20260924 20:17:38.814843 12395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:38.815018 12395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:38.815110 12395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:38.815228 12395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:38.816265 12395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:38.816632 12395 leader_election.cc:302] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e9bb2789d57410c9d1831638f6b569a; no voters:
I20260924 20:17:38.817009 12395 leader_election.cc:288] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:17:38.817121 12400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:38.817350 12400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 LEADER]: Becoming Leader. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Running, Role: LEADER
I20260924 20:17:38.817893 12400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:38.818516 12395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:17:38.820065 12401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:17:38.820190 12401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:38.820591 12409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:17:38.820570 12402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e9bb2789d57410c9d1831638f6b569a. Latest consensus state: current_term: 1 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:17:38.820786 12402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:17:38.821060 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:38.805931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.29.83.190:36969)
0924 20:17:38.806359 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:38.806375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:38.806411 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:17:38.807145 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:38.807163 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:38.807571 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:38.807863 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:38.808704 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:38.810197 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:38.811173 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:38.811941 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:38.812139 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:38.814847 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:38.814855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:38.814876 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:38.814912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:38.817014 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:38.817599 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:38.817605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:38.817612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:38.817785 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:38.818238 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:38.818241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:38.818243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:38.818847 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:38.819068 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:38.819563 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 20:17:38.819937 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260924 20:17:38.824026 12409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:17:38.829715 12409 catalog_manager.cc:1347] Generated new cluster ID: a083567d45cb459081608937fd7275f2
I20260924 20:17:38.829814 12409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:17:38.872589 12409 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:17:38.874044 12409 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:17:38.883908 12409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Generated new TSK 0
I20260924 20:17:38.884678 12409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:17:38.925263 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:39.040390 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:39.040721 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:39.040802 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:39.044974 12421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:39.045094 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:39.045192 12421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:17:39.050896 12421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:39.052132 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:39.053179 12421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:39.063169 12426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:39.063169 12427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:39.063338 12421 server_base.cc:1034] running on GCE node
W20260924 20:17:39.063174 12429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:39.064034 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:39.064733 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:39.065905 12421 hybrid_clock.cc:648] HybridClock initialized: now 1790281059065889 us; error 28 us; skew 500 ppm
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:39.068804 12421 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:17:39.069829 12421 webserver.cc:460] Webserver started at http://127.29.83.129:44367/ using document root <none> and password file <none>
I20260924 20:17:39.070380 12421 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:39.070439 12421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:39.070602 12421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:39.072616 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "cb4a8018fc1db9aa77739bc9b02c5db6"
server_key_iv: "1c07b7df188ab81fa901ac4708411386"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.073132 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "cb4a8018fc1db9aa77739bc9b02c5db6"
server_key_iv: "1c07b7df188ab81fa901ac4708411386"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.076860 12421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:17:39.079744 12436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.081275 12421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:17:39.081393 12421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "cb4a8018fc1db9aa77739bc9b02c5db6"
server_key_iv: "1c07b7df188ab81fa901ac4708411386"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.081574 12421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:39.107098 12421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:39.110972 12421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:39.111199 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:39.111860 12421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:39.112748 12421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:39.112823 12421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.112892 12421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:39.112937 12421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.122668 12421 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45991
I20260924 20:17:39.122685 12549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45991 every 8 connection(s)
I20260924 20:17:39.123637 12421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:17:39.133212 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12421
I20260924 20:17:39.133365 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:39.133678 30030 external_mini_cluster.cc:1428] Setting key e160aa32d63793805d59b1e39a06779c
I20260924 20:17:39.138945 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.125604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:53137 (local address 127.29.83.190:36969)
0924 20:17:39.125792 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.125797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.126753 (+ 956us) server_negotiation.cc:407] Connection header received
0924 20:17:39.128028 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.128031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.128105 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.128185 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:39.129477 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.130114 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.131260 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.131483 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.131570 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.134347 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:39.134351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:39.134353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:39.134382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:39.136296 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.137003 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.137012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.137017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.137124 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.137583 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.137586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.137587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.137868 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:39.137963 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.138688 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.138809 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:17:39.139832 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.126096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:53137)
0924 20:17:39.126574 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.126619 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.127821 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.128408 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.128415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.128809 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:39.129322 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.129336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.130236 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.130239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.131131 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.131140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.131629 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.131632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.131797 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.132367 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:39.132383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:39.134160 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:39.136422 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.136429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.136440 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.136723 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.137243 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.137248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.137252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.137459 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.137986 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:39.137995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:39.138384 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.138624 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.138907 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260924 20:17:39.141291 12550 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:17:39.141605 12550 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:39.142346 12550 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:17:39.144147 12359 ts_manager.cc:194] Registered new tserver with Master: 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991)
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:17:39.145402 12359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:53137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:17:39.165894 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:39.277976 12555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:39.278251 12555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:39.278295 12555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:39.282390 12555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:39.282450 12555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:39.282538 12555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:17:39.287945 12555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:39.289060 12555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:39.290175 12555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:39.300060 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:39.300060 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:39.300065 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:39.300401 12555 server_base.cc:1034] running on GCE node
I20260924 20:17:39.300809 12555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:39.301512 12555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:39.302711 12555 hybrid_clock.cc:648] HybridClock initialized: now 1790281059302685 us; error 52 us; skew 500 ppm
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:39.305922 12555 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:17:39.306939 12555 webserver.cc:460] Webserver started at http://127.29.83.130:33567/ using document root <none> and password file <none>
I20260924 20:17:39.307546 12555 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:39.307639 12555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:39.307888 12555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:39.310302 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "ef713caaba4640bcae0267dbea65a49a"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "f36c3d794915c6fd4be677c573efe0be"
server_key_iv: "3686dcab2f1724365569f3a0a0a21a94"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.310922 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "ef713caaba4640bcae0267dbea65a49a"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "f36c3d794915c6fd4be677c573efe0be"
server_key_iv: "3686dcab2f1724365569f3a0a0a21a94"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.314882 12555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:17:39.317480 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.319352 12555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:39.319502 12555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "ef713caaba4640bcae0267dbea65a49a"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "f36c3d794915c6fd4be677c573efe0be"
server_key_iv: "3686dcab2f1724365569f3a0a0a21a94"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.319646 12555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:39.339802 12555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:39.344317 12555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:39.344530 12555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:39.345136 12555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:39.346022 12555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:39.346091 12555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.346158 12555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:39.346215 12555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.355223 12555 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:42705
I20260924 20:17:39.355240 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:42705 every 8 connection(s)
I20260924 20:17:39.356110 12555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:17:39.362660 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12555
I20260924 20:17:39.362762 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:17:39.363132 30030 external_mini_cluster.cc:1428] Setting key d9461753633fecd761cc5def59c5ca94
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:39.371411 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.358322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:44083 (local address 127.29.83.190:36969)
0924 20:17:39.358476 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.358480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.359328 (+ 848us) server_negotiation.cc:407] Connection header received
0924 20:17:39.360730 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.360733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.360793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.360907 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:39.362183 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.362782 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.364242 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.364481 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.364599 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.367201 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:39.367205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:39.367210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:39.367240 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:39.368742 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.369295 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.369301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.369305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.369409 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.369974 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.369981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.369985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.370295 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:39.370426 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.371010 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.371200 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:17:39.372159 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.358710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:44083)
0924 20:17:39.359170 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.359213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.360477 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.361081 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.361089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.361451 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:39.362031 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.362046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.362969 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.362974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.364101 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.364111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.364642 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.364646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.364824 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.365455 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:39.365472 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:39.367016 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:39.368891 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.368896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.368911 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.369185 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.369535 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.369538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.369540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.369716 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.370420 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:39.370426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:39.370729 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.370903 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.371193 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:17:39.373528 12684 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:17:39.373889 12684 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:39.374413 12684 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:17:39.375640 12359 ts_manager.cc:194] Registered new tserver with Master: ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:17:39.376435 12359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:44083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:17:39.393688 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:39.508142 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:39.508431 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:39.508505 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:39.514720 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:39.514830 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:39.514932 12689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:17:39.521413 12689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:39.522938 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:39.524281 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:39.534934 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:39.534967 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:39.534924 12697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:39.535254 12689 server_base.cc:1034] running on GCE node
I20260924 20:17:39.535719 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:39.536432 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:39.537643 12689 hybrid_clock.cc:648] HybridClock initialized: now 1790281059537629 us; error 36 us; skew 500 ppm
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:39.540454 12689 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:17:39.541399 12689 webserver.cc:460] Webserver started at http://127.29.83.131:37425/ using document root <none> and password file <none>
I20260924 20:17:39.542026 12689 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:39.542092 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:39.542305 12689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:17:39.544575 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "5cdc378b11b14b8fa528a725fda67456"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "3aeabb3ff5bcf353874b0294af19c1f6"
server_key_iv: "754c4d0102600adb35f37d325eb99ece"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.545229 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "5cdc378b11b14b8fa528a725fda67456"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "3aeabb3ff5bcf353874b0294af19c1f6"
server_key_iv: "754c4d0102600adb35f37d325eb99ece"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.549515 12689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 20:17:39.552304 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.553820 12689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:17:39.553978 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "5cdc378b11b14b8fa528a725fda67456"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "3aeabb3ff5bcf353874b0294af19c1f6"
server_key_iv: "754c4d0102600adb35f37d325eb99ece"
server_key_version: "encryptionkey@0"
I20260924 20:17:39.554132 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:39.575008 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:39.578483 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:39.578719 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:39.579377 12689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:17:39.580281 12689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:17:39.580359 12689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.580439 12689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:17:39.580494 12689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:39.591070 12689 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:41289
I20260924 20:17:39.591087 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:41289 every 8 connection(s)
I20260924 20:17:39.592147 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:17:39.601974 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12689
I20260924 20:17:39.602124 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:17:39.602492 30030 external_mini_cluster.cc:1428] Setting key 10c09115df96d979ad6128be8533ebdc
Sep 24 20:17:39 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281059, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:17:39.607985 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.594433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:43669 (local address 127.29.83.190:36969)
0924 20:17:39.594640 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.594644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.595520 (+ 876us) server_negotiation.cc:407] Connection header received
0924 20:17:39.596934 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.596937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.596999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.597110 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:39.598412 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.599157 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.600894 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.601121 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.601251 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.604603 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:39.604608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:39.604610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:39.604646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:39.606084 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.606629 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.606631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.606633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.606728 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.607016 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.607019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.607020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.607192 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:39.607264 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.607767 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.607867 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 20:17:39.608858 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.594801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:43669)
0924 20:17:39.595320 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.595377 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.596688 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.597305 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.597312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.597710 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:17:39.598244 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.598258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.599329 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.599343 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.600767 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.600779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.601350 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.601357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.601642 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.602443 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:17:39.602469 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:17:39.604436 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:17:39.606209 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.606214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.606228 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.606509 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.606808 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:17:39.606810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:17:39.606812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:17:39.606946 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:17:39.607281 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:17:39.607286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:17:39.607527 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.607718 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.607977 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260924 20:17:39.610189 12818 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:17:39.610447 12818 heartbeater.cc:464] Registering TS with master...
I20260924 20:17:39.610935 12818 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:17:39.612007 12359 ts_manager.cc:194] Registered new tserver with Master: 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289)
I20260924 20:17:39.612618 12359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:43669
I20260924 20:17:39.616765 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:17:39.626703 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.618935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.29.83.190:36969)
0924 20:17:39.619054 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.619061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.619264 (+ 203us) server_negotiation.cc:407] Connection header received
0924 20:17:39.619431 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.619434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.619483 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.619566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:17:39.620298 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.620897 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.621738 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.622000 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.622112 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.623539 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:17:39.623543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:17:39.623545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:17:39.623570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:17:39.625005 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.625430 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.625435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.625437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.625517 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:17:39.625816 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:17:39.625820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:17:39.625825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:17:39.626047 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:17:39.626196 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.626425 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.626542 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:17:39.629563 12359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:17:39.632669 12359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:17:39.647812 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.642396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41289 (local address 127.0.0.1:39360)
0924 20:17:39.642780 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.642838 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.643144 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.643706 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.643711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.643733 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:39.643889 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.643896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.644974 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.644980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.646314 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.646323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.647252 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.647256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.647362 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.647403 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.647520 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.647650 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":22,"thread_start_us":103,"threads_started":1}
I20260924 20:17:39.647840 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.642035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.29.83.131:41289)
0924 20:17:39.642411 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.642425 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.642926 (+ 501us) server_negotiation.cc:407] Connection header received
0924 20:17:39.643337 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.643345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.643551 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.643713 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:39.644021 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.644837 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.646438 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.647127 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.647237 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.647385 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.647573 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.647675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260924 20:17:39.648512 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.641677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45991 (local address 127.0.0.1:43578)
0924 20:17:39.642780 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.642838 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.643052 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.643825 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.643828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.643846 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:39.644024 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.644033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.645514 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.645518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.646770 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.646781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.648054 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.648057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.648151 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.648201 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.648290 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.648361 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":46,"thread_start_us":110,"threads_started":1}
I20260924 20:17:39.648890 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.642858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.29.83.130:42705)
0924 20:17:39.643177 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.643193 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.643382 (+ 189us) server_negotiation.cc:407] Connection header received
0924 20:17:39.643458 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.643462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.643632 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.643764 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:39.644219 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.645362 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.647004 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.647829 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.647942 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.648102 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.648631 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.648735 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260924 20:17:39.649955 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.641866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.29.83.129:45991)
0924 20:17:39.642639 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.642646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.642926 (+ 280us) server_negotiation.cc:407] Connection header received
0924 20:17:39.643263 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.643269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.643499 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.643626 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:39.644166 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.645362 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.646916 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.647729 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.647829 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.647980 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.648787 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.648878 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":94,"threads_started":1}
I20260924 20:17:39.649281 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.642421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42705 (local address 127.0.0.1:59836)
0924 20:17:39.643251 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.643265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.643367 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.643876 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.643880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.643895 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:39.644061 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.644069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.645728 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.645731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.646882 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.646892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.648164 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.648167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.648317 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.648342 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.648422 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.648492 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":52,"thread_start_us":82,"threads_started":1}
I20260924 20:17:39.651221 12752 tablet_service.cc:1467] Processing CreateTablet for tablet a2eabf56091b4e26b481e64f1dc17719 (DEFAULT_TABLE table=test-table [id=1c10aec418034fc0852d0e0326eb48ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:39.652412 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2eabf56091b4e26b481e64f1dc17719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:39.652770 12618 tablet_service.cc:1467] Processing CreateTablet for tablet a2eabf56091b4e26b481e64f1dc17719 (DEFAULT_TABLE table=test-table [id=1c10aec418034fc0852d0e0326eb48ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:39.653353 12484 tablet_service.cc:1467] Processing CreateTablet for tablet a2eabf56091b4e26b481e64f1dc17719 (DEFAULT_TABLE table=test-table [id=1c10aec418034fc0852d0e0326eb48ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:17:39.653739 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2eabf56091b4e26b481e64f1dc17719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:39.654345 12484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2eabf56091b4e26b481e64f1dc17719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:17:39.659821 12835 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Bootstrap starting.
I20260924 20:17:39.661089 12836 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Bootstrap starting.
I20260924 20:17:39.661193 12837 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap starting.
I20260924 20:17:39.662597 12835 tablet_bootstrap.cc:654] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:39.663429 12837 tablet_bootstrap.cc:654] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:39.663489 12836 tablet_bootstrap.cc:654] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Neither blocks nor log segments found. Creating new log.
I20260924 20:17:39.663654 12835 log.cc:824] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:39.664279 12837 log.cc:824] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:39.664374 12836 log.cc:824] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:39.665721 12835 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: No bootstrap required, opened a new log
I20260924 20:17:39.665963 12835 ts_tablet_manager.cc:1397] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 20:17:39.666296 12836 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: No bootstrap required, opened a new log
I20260924 20:17:39.666296 12837 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: No bootstrap required, opened a new log
I20260924 20:17:39.666479 12837 ts_tablet_manager.cc:1397] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:17:39.666479 12836 ts_tablet_manager.cc:1397] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:17:39.668910 12835 raft_consensus.cc:348] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.669178 12835 raft_consensus.cc:374] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:39.669260 12835 raft_consensus.cc:729] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef713caaba4640bcae0267dbea65a49a, State: Initialized, Role: FOLLOWER
I20260924 20:17:39.669817 12835 consensus_queue.cc:260] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.670516 12836 raft_consensus.cc:348] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.670811 12836 raft_consensus.cc:374] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:39.670869 12836 raft_consensus.cc:729] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdc378b11b14b8fa528a725fda67456, State: Initialized, Role: FOLLOWER
I20260924 20:17:39.670912 12837 raft_consensus.cc:348] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.671152 12835 ts_tablet_manager.cc:1428] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 20:17:39.671191 12837 raft_consensus.cc:374] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:17:39.671271 12837 raft_consensus.cc:729] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ceeba2f3c46dfb4a09a662f79ba69, State: Initialized, Role: FOLLOWER
I20260924 20:17:39.671314 12836 consensus_queue.cc:260] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.671511 12684 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
I20260924 20:17:39.671717 12837 consensus_queue.cc:260] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.671984 12818 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
I20260924 20:17:39.672334 12836 ts_tablet_manager.cc:1428] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:17:39.672797 12837 ts_tablet_manager.cc:1428] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:17:39.673142 12550 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
I20260924 20:17:39.683912 12843 raft_consensus.cc:482] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:17:39.684092 12843 raft_consensus.cc:504] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.685300 12843 leader_election.cc:288] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
I20260924 20:17:39.689440 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.685692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:36173 (local address 127.29.83.130:42705)
0924 20:17:39.685840 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.685845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.686092 (+ 247us) server_negotiation.cc:407] Connection header received
0924 20:17:39.686265 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.686268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.686320 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.686404 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:39.686704 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.687253 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.688404 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.688887 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.689002 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.689069 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.689249 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.689318 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:17:39.689471 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.685542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42705 (local address 127.29.83.129:36173)
0924 20:17:39.685990 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.686002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.686136 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.686412 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.686415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.686453 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:39.686592 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.686598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.687364 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.687369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.688284 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.688293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.689008 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.689012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.689100 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.689131 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.689230 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.689303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":55,"spinlock_wait_cycles":4736,"thread_start_us":178,"threads_started":1}
I20260924 20:17:39.689594 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.685603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:35051 (local address 127.29.83.131:41289)
0924 20:17:39.685751 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.685783 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.686072 (+ 289us) server_negotiation.cc:407] Connection header received
0924 20:17:39.686237 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.686244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.686284 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.686362 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:17:39.686647 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.687193 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.688407 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.688913 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.689043 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.689098 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.689391 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.689465 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:17:39.689963 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.685570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41289 (local address 127.29.83.129:35051)
0924 20:17:39.685977 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.685989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:17:39.686132 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:17:39.686371 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:17:39.686373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:17:39.686388 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:17:39.686546 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.686551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.687292 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.687298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.688290 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:17:39.688300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.689149 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.689152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:17:39.689244 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.689270 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:17:39.689737 (+ 467us) client_negotiation.cc:240] Negotiation successful
0924 20:17:39.689824 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":85,"spinlock_wait_cycles":13696,"thread_start_us":125,"threads_started":1}
I20260924 20:17:39.690339 12638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2eabf56091b4e26b481e64f1dc17719" candidate_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef713caaba4640bcae0267dbea65a49a" is_pre_election: true
I20260924 20:17:39.690544 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2eabf56091b4e26b481e64f1dc17719" candidate_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdc378b11b14b8fa528a725fda67456" is_pre_election: true
I20260924 20:17:39.690622 12638 raft_consensus.cc:2454] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971ceeba2f3c46dfb4a09a662f79ba69 in term 0.
I20260924 20:17:39.690762 12772 raft_consensus.cc:2454] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971ceeba2f3c46dfb4a09a662f79ba69 in term 0.
I20260924 20:17:39.691093 12438 leader_election.cc:302] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69, ef713caaba4640bcae0267dbea65a49a; no voters:
I20260924 20:17:39.691390 12843 raft_consensus.cc:2781] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:17:39.691476 12843 raft_consensus.cc:482] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:17:39.691529 12843 raft_consensus.cc:3037] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:39.692520 12843 raft_consensus.cc:504] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.692895 12843 leader_election.cc:288] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 1 election: Requested vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
I20260924 20:17:39.693192 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2eabf56091b4e26b481e64f1dc17719" candidate_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdc378b11b14b8fa528a725fda67456"
I20260924 20:17:39.693234 12638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2eabf56091b4e26b481e64f1dc17719" candidate_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef713caaba4640bcae0267dbea65a49a"
I20260924 20:17:39.693320 12772 raft_consensus.cc:3037] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:39.693334 12638 raft_consensus.cc:3037] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:17:39.694603 12772 raft_consensus.cc:2454] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971ceeba2f3c46dfb4a09a662f79ba69 in term 1.
I20260924 20:17:39.694603 12638 raft_consensus.cc:2454] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971ceeba2f3c46dfb4a09a662f79ba69 in term 1.
I20260924 20:17:39.694908 12440 leader_election.cc:302] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cdc378b11b14b8fa528a725fda67456, 971ceeba2f3c46dfb4a09a662f79ba69; no voters:
I20260924 20:17:39.695109 12843 raft_consensus.cc:2781] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:17:39.695345 12843 raft_consensus.cc:686] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 LEADER]: Becoming Leader. State: Replica: 971ceeba2f3c46dfb4a09a662f79ba69, State: Running, Role: LEADER
I20260924 20:17:39.695680 12843 consensus_queue.cc:237] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:17:39.698954 12358 catalog_manager.cc:5495] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129). New cstate: current_term: 1 leader_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } health_report { overall_health: UNKNOWN } } }
I20260924 20:17:39.710583 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.706795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.29.83.129:45991)
0924 20:17:39.706930 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:17:39.706934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:17:39.706947 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:17:39.707173 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:17:39.707176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:17:39.707234 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:17:39.707335 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:17:39.707655 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.708176 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.709066 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:17:39.709283 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:17:39.709375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:17:39.709786 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:17:39.710212 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:17:39.710310 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:17:39.710413 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 20:17:39.710466 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:17:39.716434 12638 raft_consensus.cc:1261] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Refusing update from remote peer 971ceeba2f3c46dfb4a09a662f79ba69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:39.716482 12772 raft_consensus.cc:1261] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Refusing update from remote peer 971ceeba2f3c46dfb4a09a662f79ba69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:17:39.717224 12846 consensus_queue.cc:1035] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:39.717465 12843 consensus_queue.cc:1035] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:17:39.729564 12850 mvcc.cc:205] Tried to move back new op lower bound from 7332991220591149056 to 7332991220516237312. Current Snapshot: MvccSnapshot[applied={T|T < 7332991220591149056 or (T in {7332991220591149056})}]
I20260924 20:17:39.735447 12357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:39.735607 12357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:17:39.738178 12357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Sending DeleteTablet for 3 replicas of tablet a2eabf56091b4e26b481e64f1dc17719
I20260924 20:17:39.739097 12752 tablet_service.cc:1514] Processing DeleteTablet for tablet a2eabf56091b4e26b481e64f1dc17719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:39360
I20260924 20:17:39.739208 12618 tablet_service.cc:1514] Processing DeleteTablet for tablet a2eabf56091b4e26b481e64f1dc17719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:59836
I20260924 20:17:39.739463 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12421
I20260924 20:17:39.739626 12484 tablet_service.cc:1514] Processing DeleteTablet for tablet a2eabf56091b4e26b481e64f1dc17719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:43578
I20260924 20:17:39.741083 12861 tablet_replica.cc:330] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: stopping tablet replica
I20260924 20:17:39.741093 12860 tablet_replica.cc:330] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: stopping tablet replica
I20260924 20:17:39.741402 12861 raft_consensus.cc:2229] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:39.741446 12860 raft_consensus.cc:2229] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:17:39.741839 12860 raft_consensus.cc:2258] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:39.741827 12861 raft_consensus.cc:2258] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:17:39.743332 12861 ts_tablet_manager.cc:1905] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:39.743430 12860 ts_tablet_manager.cc:1905] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:17:39.745535 12861 ts_tablet_manager.cc:1918] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:39.745645 12861 log.cc:1196] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/a2eabf56091b4e26b481e64f1dc17719
I20260924 20:17:39.746172 12861 ts_tablet_manager.cc:1939] T a2eabf56091b4e26b481e64f1dc17719 P ef713caaba4640bcae0267dbea65a49a: Deleting consensus metadata
W20260924 20:17:39.746646 12345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:45991 (error 108)
I20260924 20:17:39.746791 12860 ts_tablet_manager.cc:1918] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:17:39.746879 12860 log.cc:1196] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/a2eabf56091b4e26b481e64f1dc17719
I20260924 20:17:39.747215 12860 ts_tablet_manager.cc:1939] T a2eabf56091b4e26b481e64f1dc17719 P 5cdc378b11b14b8fa528a725fda67456: Deleting consensus metadata
I20260924 20:17:39.747602 12344 catalog_manager.cc:4841] TS ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): tablet a2eabf56091b4e26b481e64f1dc17719 (table test-table [id=1c10aec418034fc0852d0e0326eb48ed]) successfully deleted
I20260924 20:17:39.748085 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12555
I20260924 20:17:39.748180 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:17:39.747891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45991 (local address 127.0.0.1:43586)
0924 20:17:39.748017 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:17:39.748084 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 20:17:39.748443 12346 catalog_manager.cc:4841] TS 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): tablet a2eabf56091b4e26b481e64f1dc17719 (table test-table [id=1c10aec418034fc0852d0e0326eb48ed]) successfully deleted
W20260924 20:17:39.748826 12345 catalog_manager.cc:4584] TS 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2eabf56091b4e26b481e64f1dc17719: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45991: connect: Connection refused (error 111)
I20260924 20:17:39.754685 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12689
I20260924 20:17:39.760718 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12327
I20260924 20:17:59.767485 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--webserver_interface=127.29.83.190
--webserver_port=34719
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:17:59.879791 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:17:59.880103 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:17:59.880174 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:17:59.884189 12863 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:17:59.884280 12863 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:17:59.884310 12863 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:17:59.884341 12863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:17:59.884380 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:17:59.884425 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:17:59.884480 12863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:17:59.884518 12863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:17:59.889990 12863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:17:59.891083 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:17:59.892139 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:17:59.906160 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:59.906260 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:17:59.906188 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:17:59.906373 12863 server_base.cc:1034] running on GCE node
I20260924 20:17:59.907171 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:17:59.908457 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:17:59.909678 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790281079909658 us; error 33 us; skew 500 ppm
Sep 24 20:17:59 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:17:59.920511 12863 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:17:59.921679 12863 webserver.cc:460] Webserver started at http://127.29.83.190:34719/ using document root <none> and password file <none>
I20260924 20:17:59.922372 12863 fs_manager.cc:359] Metadata directory not provided
I20260924 20:17:59.922500 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:17:59.926856 12863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:17:59.929282 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:17:59.930842 12863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:17:59.930968 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5e9bb2789d57410c9d1831638f6b569a"
format_stamp: "Formatted at 2026-09-24 20:17:38 on dist-test-slave-52fr"
server_key: "6aa52da0019bc7495d67e9493d5059ab"
server_key_iv: "54bdfeccf1253933fd092558e78a4a5c"
server_key_version: "encryptionkey@0"
I20260924 20:17:59.931607 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:17:59.944101 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:17:59.947885 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:17:59.948103 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:17:59.955862 12863 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36969
I20260924 20:17:59.956154 12931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36969 every 8 connection(s)
I20260924 20:17:59.957008 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:17:59.961452 12932 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:17:59.963200 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap starting.
I20260924 20:17:59.964848 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12863
I20260924 20:17:59.965349 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:45991
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=44367
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:17:59.981586 12932 log.cc:824] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:17:59.990483 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:17:59.990867 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap complete.
I20260924 20:17:59.995380 12932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:59.995859 12932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Initialized, Role: FOLLOWER
I20260924 20:17:59.996419 12932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:59.996569 12932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:17:59.996666 12932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:17:59.996783 12932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:17:59.998857 12932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:17:59.999202 12932 leader_election.cc:302] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e9bb2789d57410c9d1831638f6b569a; no voters:
I20260924 20:17:59.999578 12932 leader_election.cc:288] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:17:59.999696 12936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:17:59.999959 12936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 LEADER]: Becoming Leader. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Running, Role: LEADER
I20260924 20:18:00.000335 12936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:18:00.000834 12932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:00.001391 12936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e9bb2789d57410c9d1831638f6b569a. Latest consensus state: current_term: 2 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:18:00.001832 12936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:00.002131 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:18:00.002871 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:00.005676 12952 catalog_manager.cc:1259] Loaded cluster ID: a083567d45cb459081608937fd7275f2
I20260924 20:18:00.005753 12952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: loading cluster ID for follower catalog manager: success
I20260924 20:18:00.007543 12952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: acquiring CA information for follower catalog manager: success
I20260924 20:18:00.008797 12952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:18:00.008951 12945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:00.011802 12945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:00.012306 12945 catalog_manager.cc:1259] Loaded cluster ID: a083567d45cb459081608937fd7275f2
I20260924 20:18:00.012370 12945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:00.013089 12945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:00.013569 12945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Loaded TSK: 0
I20260924 20:18:00.025671 12945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Generated new TSK 1
I20260924 20:18:00.025975 12945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:18:00.090487 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:00.090785 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:00.090832 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:00.095052 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:00.095129 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:00.095216 12934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:00.100620 12934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:45991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=44367
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:00.101907 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:00.102986 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:00.114343 12959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:00.114480 12962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:00.114341 12960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:00.115242 12934 server_base.cc:1034] running on GCE node
I20260924 20:18:00.115726 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:00.116479 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:00.117687 12934 hybrid_clock.cc:648] HybridClock initialized: now 1790281080117666 us; error 36 us; skew 500 ppm
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:00.120601 12934 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:18:00.121671 12934 webserver.cc:460] Webserver started at http://127.29.83.129:44367/ using document root <none> and password file <none>
I20260924 20:18:00.122269 12934 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:00.122341 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:00.126336 12934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:00.128537 12969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.129972 12934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:00.130100 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "cb4a8018fc1db9aa77739bc9b02c5db6"
server_key_iv: "1c07b7df188ab81fa901ac4708411386"
server_key_version: "encryptionkey@0"
I20260924 20:18:00.130673 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:00.145531 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:00.149104 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:00.149340 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:00.150080 12934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:00.151211 12976 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:18:00.155401 12934 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:18:00.155473 12934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 20:18:00.155575 12934 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:18:00.157289 12934 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:18:00.157337 12934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:18:00.157644 12976 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap starting.
I20260924 20:18:00.167933 12934 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45991
I20260924 20:18:00.168001 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45991 every 8 connection(s)
I20260924 20:18:00.168946 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:00.172525 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 12934
I20260924 20:18:00.173192 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:42705
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=33567
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:18:00.176916 12976 log.cc:824] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:00.185225 12976 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:18:00.185628 12976 tablet_bootstrap.cc:492] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap complete.
I20260924 20:18:00.187031 12976 ts_tablet_manager.cc:1397] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20260924 20:18:00.189574 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.171234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:43833)
0924 20:18:00.171843 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.171895 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.172813 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.174158 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.174166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.174608 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:00.175338 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.175361 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.177377 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.177384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.178997 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.179028 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.180057 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.180065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.180309 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.181079 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:00.181109 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:00.183151 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:00.186047 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.186056 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.186073 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.186448 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.186863 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.186869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.186874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.187055 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.187764 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:00.187770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:00.188070 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.188284 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.188624 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":136,"threads_started":1}
I20260924 20:18:00.189664 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.171293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:43833 (local address 127.29.83.190:36969)
0924 20:18:00.171730 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.171752 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.172022 (+ 270us) server_negotiation.cc:407] Connection header received
0924 20:18:00.173247 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.173258 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.173687 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.174095 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.175563 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.177195 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.179167 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.179874 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.180066 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.183396 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.183407 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.183428 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.183475 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.185912 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.186580 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.186586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.186592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.186750 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.187180 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.187184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.187186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.187637 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.187772 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.188433 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.188735 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260924 20:18:00.190007 12976 raft_consensus.cc:348] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:18:00.190511 12976 raft_consensus.cc:729] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ceeba2f3c46dfb4a09a662f79ba69, State: Initialized, Role: FOLLOWER
I20260924 20:18:00.191030 12976 consensus_queue.cc:260] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } } peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } }
I20260924 20:18:00.192085 12976 ts_tablet_manager.cc:1428] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:18:00.192695 13084 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:18:00.192899 13084 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:00.193331 13084 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:18:00.195863 12896 ts_manager.cc:194] Registered new tserver with Master: 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991)
I20260924 20:18:00.200013 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:43833
I20260924 20:18:00.202950 13084 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.129@KRBTEST.COM
I20260924 20:18:00.210986 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.199372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45991 (local address 127.0.0.1:41928)
0924 20:18:00.200237 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.200286 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.200429 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.201136 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.201142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.201582 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:00.201693 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.201699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.202826 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.202831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.203722 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.203731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.204310 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.204314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.204430 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.205202 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:00.205215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:00.206716 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:00.209006 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.209016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.209023 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.209269 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.209643 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.209646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.209649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.209744 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.210275 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:00.210281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:00.210413 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.210550 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.210776 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":101,"threads_started":1}
I20260924 20:18:00.210986 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.199505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.29.83.129:45991)
0924 20:18:00.199927 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.199938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.200634 (+ 696us) server_negotiation.cc:407] Connection header received
0924 20:18:00.200718 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.200724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.200974 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.201136 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.201904 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.202712 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.203836 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.204180 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.204295 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.206880 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.206889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.206900 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.206923 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.208866 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.209418 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.209445 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.209447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.209532 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.209872 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.209878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.209880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.210148 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.210292 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.210650 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.210820 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":104,"threads_started":1}
I20260924 20:18:00.212217 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet a2eabf56091b4e26b481e64f1dc17719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:17:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:41928
I20260924 20:18:00.213122 13093 tablet_replica.cc:330] stopping tablet replica
I20260924 20:18:00.213424 13093 raft_consensus.cc:2229] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:00.213694 13093 raft_consensus.cc:2258] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:00.214721 13093 ts_tablet_manager.cc:1905] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:00.218448 13093 ts_tablet_manager.cc:1918] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:00.218542 13093 log.cc:1196] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal/wals/a2eabf56091b4e26b481e64f1dc17719
I20260924 20:18:00.218815 13093 ts_tablet_manager.cc:1939] T a2eabf56091b4e26b481e64f1dc17719 P 971ceeba2f3c46dfb4a09a662f79ba69: Deleting consensus metadata
I20260924 20:18:00.221035 12882 catalog_manager.cc:4841] TS 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991): tablet a2eabf56091b4e26b481e64f1dc17719 (table test-table [id=1c10aec418034fc0852d0e0326eb48ed]) successfully deleted
W20260924 20:18:00.300407 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:00.300755 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:00.300841 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:00.305331 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:00.305425 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:00.305526 13088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:00.311169 13088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:42705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=33567
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:00.312230 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:00.313199 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:00.322870 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:00.322865 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:00.323177 13088 server_base.cc:1034] running on GCE node
W20260924 20:18:00.322877 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:00.323702 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:00.324385 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:00.325563 13088 hybrid_clock.cc:648] HybridClock initialized: now 1790281080325548 us; error 26 us; skew 500 ppm
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:00.328670 13088 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:18:00.329810 13088 webserver.cc:460] Webserver started at http://127.29.83.130:33567/ using document root <none> and password file <none>
I20260924 20:18:00.330533 13088 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:00.330627 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:00.336380 13088 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:00.338876 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.340389 13088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:00.340561 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "ef713caaba4640bcae0267dbea65a49a"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "f36c3d794915c6fd4be677c573efe0be"
server_key_iv: "3686dcab2f1724365569f3a0a0a21a94"
server_key_version: "encryptionkey@0"
I20260924 20:18:00.341174 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:00.368223 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:00.371658 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:00.371868 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:00.372483 13088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:00.373340 13088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:00.373413 13088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.373484 13088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:00.373533 13088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.383715 13088 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:42705
I20260924 20:18:00.383733 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:42705 every 8 connection(s)
I20260924 20:18:00.384840 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:00.393051 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13088
I20260924 20:18:00.393628 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:41289
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=37425
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:00.400310 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.386769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:34043 (local address 127.29.83.190:36969)
0924 20:18:00.386955 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.386960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.387836 (+ 876us) server_negotiation.cc:407] Connection header received
0924 20:18:00.389135 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.389137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.389185 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.389259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.390563 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.391115 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.392489 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.392774 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.392868 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.396391 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.396395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.396397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.396428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.398146 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.398693 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.398697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.398699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.398792 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.399145 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.399149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.399151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.399352 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.399461 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.399998 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.400143 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:18:00.401197 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.387209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:34043)
0924 20:18:00.387663 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.387718 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.388904 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.389432 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.389440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.389906 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:00.390407 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.390421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.391266 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.391270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.392338 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.392350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.392960 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.392965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.393220 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.394155 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:00.394183 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:00.396144 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:00.398269 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.398276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.398294 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.398581 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.398887 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.398892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.398894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.399045 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.399460 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:00.399477 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:00.399748 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.399950 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.400316 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260924 20:18:00.402709 13222 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:18:00.403123 13222 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:00.403702 13222 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:18:00.404850 12896 ts_manager.cc:194] Registered new tserver with Master: ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
I20260924 20:18:00.405619 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:34043
W20260924 20:18:00.522329 13225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:00.522614 13225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:00.522662 13225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:00.527261 13225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:00.527325 13225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:00.527387 13225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:00.532836 13225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:41289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=37425
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:00.533946 13225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:00.535023 13225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:00.545109 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:00.545185 13231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:00.545369 13233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:00.546083 13225 server_base.cc:1034] running on GCE node
I20260924 20:18:00.546497 13225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:00.547137 13225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:00.548360 13225 hybrid_clock.cc:648] HybridClock initialized: now 1790281080548344 us; error 28 us; skew 500 ppm
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:00.551610 13225 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:18:00.552562 13225 webserver.cc:460] Webserver started at http://127.29.83.131:37425/ using document root <none> and password file <none>
I20260924 20:18:00.553153 13225 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:00.553270 13225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:00.557367 13225 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:00.559567 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.560786 13225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:00.560937 13225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "5cdc378b11b14b8fa528a725fda67456"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "3aeabb3ff5bcf353874b0294af19c1f6"
server_key_iv: "754c4d0102600adb35f37d325eb99ece"
server_key_version: "encryptionkey@0"
I20260924 20:18:00.561515 13225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:00.591648 13225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:00.595144 13225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:00.595357 13225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:00.595981 13225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:00.596822 13225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:00.596890 13225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.596962 13225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:00.597013 13225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:00.607406 13225 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:41289
I20260924 20:18:00.607424 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:41289 every 8 connection(s)
I20260924 20:18:00.608532 13225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:00.611995 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13225
Sep 24 20:18:00 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281080, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:00.622633 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.612711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.29.83.190:36969)
0924 20:18:00.612959 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.612963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.613139 (+ 176us) server_negotiation.cc:407] Connection header received
0924 20:18:00.613265 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.613271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.613335 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.613427 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.614383 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.615128 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.616247 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.616484 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.616563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.618235 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.618239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.618242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.618268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.620011 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.620483 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.620488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.620491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.620589 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.620978 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.620984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.620986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.621334 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.621561 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.621983 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.622326 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"spinlock_wait_cycles":5760,"thread_start_us":62,"threads_started":1}
I20260924 20:18:00.624280 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.610451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:33685 (local address 127.29.83.190:36969)
0924 20:18:00.610621 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.610625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.611451 (+ 826us) server_negotiation.cc:407] Connection header received
0924 20:18:00.613107 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.613129 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.613185 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.613282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.614793 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.615431 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.616882 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.617077 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.617183 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.620087 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.620093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.620096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.620123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.621881 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.622516 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.622520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.622521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.622629 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.623070 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.623074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.623075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.623283 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.623356 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.624021 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.624131 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"spinlock_wait_cycles":31232}
I20260924 20:18:00.625216 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.610833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:33685)
0924 20:18:00.611291 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.611335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.612840 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.613502 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.613511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.614069 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:00.614649 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.614662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.615546 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.615549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.616749 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.616762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.617235 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.617240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.617420 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.618030 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:00.618047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:00.619884 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:00.622012 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.622017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.622032 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.622403 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.622763 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:00.622767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:00.622769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:00.622950 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:00.623438 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:00.623444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:00.623760 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.623961 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.624278 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260924 20:18:00.626482 13354 heartbeater.cc:347] Connected to a master server at 127.29.83.190:36969
I20260924 20:18:00.626732 13354 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:00.627259 13354 heartbeater.cc:511] Master 127.29.83.190:36969 requested a full tablet report, sending...
I20260924 20:18:00.628301 12896 ts_manager.cc:194] Registered new tserver with Master: 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289)
I20260924 20:18:00.628845 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:33685
I20260924 20:18:00.636544 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:00.640734 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.637169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.29.83.190:36969)
0924 20:18:00.637309 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.637315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.637505 (+ 190us) server_negotiation.cc:407] Connection header received
0924 20:18:00.637697 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.637700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.637843 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.637950 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:00.638199 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.638707 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.639580 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.639794 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.639898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.640203 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:00.640334 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 20:18:00.640597 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56798: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 20:18:00.640794 13086 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56798: token expired
W20260924 20:18:00.641206 12821 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.83.190:36969)
I20260924 20:18:00.648610 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.641611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.29.83.190:36969)
0924 20:18:00.641753 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.641796 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.641815 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:18:00.641936 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.641939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.641980 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.642052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:00.642602 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.643080 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.643862 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.644080 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.644166 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.645473 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:00.645479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:00.645481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:00.645503 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:00.647028 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.647421 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.647424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.647426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.647505 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:00.647731 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:00.647734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:00.647735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:00.647893 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:00.648009 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.648264 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.648454 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:18:00.651110 12896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:00.654767 12896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:00.662662 13017 tablet_service.cc:1467] Processing CreateTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd (DEFAULT_TABLE table=test-table [id=d7c0c606ef154840a01ba17c4d99aed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:00.663326 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c0581e22a84764b8f837ea0e8c07dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:00.666289 12976 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap starting.
I20260924 20:18:00.667256 12976 tablet_bootstrap.cc:654] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:00.667668 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.661649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41289 (local address 127.0.0.1:54330)
0924 20:18:00.661823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.661838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.661972 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.663267 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.663270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.663288 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:00.663403 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.663419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.664859 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.664865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.665987 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.666002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.667223 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.667226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.667325 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.667373 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.667438 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.667533 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":3}
I20260924 20:18:00.667980 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.662472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.29.83.131:41289)
0924 20:18:00.662737 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.662751 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.662776 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:18:00.662860 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.662867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.663100 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.663273 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:00.663560 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.664710 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.666148 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.667102 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.667224 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.667479 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.667533 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.667622 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260924 20:18:00.668016 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.662148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42705 (local address 127.0.0.1:45596)
0924 20:18:00.662449 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.662463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.662551 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.663511 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.663514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.663552 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:00.663646 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.663654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.664885 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.664887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.666181 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.666195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.667536 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.667542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.667683 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.667709 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.667788 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.667854 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":117,"thread_start_us":119,"threads_started":1}
I20260924 20:18:00.668576 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.662302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.29.83.130:42705)
0924 20:18:00.662850 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.662861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.662879 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:18:00.662970 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.662975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.663116 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.663273 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:00.663791 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.664701 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.666331 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.667172 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.667284 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.667457 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.668293 (+ 836us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.668402 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":90,"threads_started":1}
I20260924 20:18:00.669282 12976 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: No bootstrap required, opened a new log
I20260924 20:18:00.669363 12976 ts_tablet_manager.cc:1397] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:18:00.669808 12976 raft_consensus.cc:348] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.669916 12976 raft_consensus.cc:374] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:00.669952 12976 raft_consensus.cc:729] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ceeba2f3c46dfb4a09a662f79ba69, State: Initialized, Role: FOLLOWER
I20260924 20:18:00.670105 12976 consensus_queue.cc:260] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.670344 12976 ts_tablet_manager.cc:1428] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 20:18:00.670585 13085 tablet.cc:2369] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:00.671180 13288 tablet_service.cc:1467] Processing CreateTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd (DEFAULT_TABLE table=test-table [id=d7c0c606ef154840a01ba17c4d99aed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:00.671460 13156 tablet_service.cc:1467] Processing CreateTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd (DEFAULT_TABLE table=test-table [id=d7c0c606ef154840a01ba17c4d99aed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:00.672329 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c0581e22a84764b8f837ea0e8c07dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:00.672317 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c0581e22a84764b8f837ea0e8c07dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:00.679147 13367 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Bootstrap starting.
I20260924 20:18:00.679824 13368 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Bootstrap starting.
I20260924 20:18:00.681656 13367 tablet_bootstrap.cc:654] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:00.681929 13368 tablet_bootstrap.cc:654] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:00.682473 13367 log.cc:824] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:00.682608 13368 log.cc:824] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:00.684218 13367 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: No bootstrap required, opened a new log
I20260924 20:18:00.684230 13368 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: No bootstrap required, opened a new log
I20260924 20:18:00.684422 13367 ts_tablet_manager.cc:1397] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:18:00.684419 13368 ts_tablet_manager.cc:1397] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:18:00.687394 13367 raft_consensus.cc:348] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.687394 13368 raft_consensus.cc:348] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.687603 13368 raft_consensus.cc:374] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:00.687633 13367 raft_consensus.cc:374] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:00.687660 13368 raft_consensus.cc:729] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdc378b11b14b8fa528a725fda67456, State: Initialized, Role: FOLLOWER
I20260924 20:18:00.687692 13367 raft_consensus.cc:729] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef713caaba4640bcae0267dbea65a49a, State: Initialized, Role: FOLLOWER
I20260924 20:18:00.688098 13368 consensus_queue.cc:260] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.688098 13367 consensus_queue.cc:260] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.688725 13354 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
I20260924 20:18:00.688725 13222 heartbeater.cc:503] Master 127.29.83.190:36969 was elected leader, sending a full tablet report...
I20260924 20:18:00.689066 13367 ts_tablet_manager.cc:1428] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:18:00.689069 13368 ts_tablet_manager.cc:1428] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 20:18:00.859572 13355 tablet.cc:2369] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:00.886116 13223 tablet.cc:2369] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:00.916028 13372 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:00.916224 13372 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.917403 13372 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705), 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991)
I20260924 20:18:00.921104 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.917802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:36339 (local address 127.29.83.130:42705)
0924 20:18:00.917924 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.917930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.917946 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:18:00.918132 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.918135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.918179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.918278 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:00.918571 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.919124 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920149 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.920542 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920668 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.920720 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.920901 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.920972 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:18:00.921104 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.917648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42705 (local address 127.29.83.131:36339)
0924 20:18:00.917862 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.917878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.917991 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.918282 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.918285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.918302 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:00.918426 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.918431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.919230 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.919236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.920023 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.920031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.920660 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.920771 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.920794 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.920886 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.920961 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1}
I20260924 20:18:00.921325 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.917663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45991 (local address 127.29.83.131:34179)
0924 20:18:00.918089 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:00.918103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:00.918239 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:00.918543 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:00.918546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:00.918561 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:00.918689 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.918695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.919400 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.919407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.920284 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:00.920294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.920947 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:00.921043 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.921065 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:18:00.921125 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:18:00.921185 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":13,"thread_start_us":134,"threads_started":1}
I20260924 20:18:00.921654 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.917747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:34179 (local address 127.29.83.129:45991)
0924 20:18:00.918071 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.918076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.918195 (+ 119us) server_negotiation.cc:407] Connection header received
0924 20:18:00.918352 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.918355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.918422 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.918535 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:00.918797 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.919293 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920403 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.920828 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.920970 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.921371 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.921415 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.921501 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":72,"threads_started":1}
I20260924 20:18:00.921833 13176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c0581e22a84764b8f837ea0e8c07dd" candidate_uuid: "5cdc378b11b14b8fa528a725fda67456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef713caaba4640bcae0267dbea65a49a" is_pre_election: true
I20260924 20:18:00.922183 13176 raft_consensus.cc:2454] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cdc378b11b14b8fa528a725fda67456 in term 0.
I20260924 20:18:00.922148 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c0581e22a84764b8f837ea0e8c07dd" candidate_uuid: "5cdc378b11b14b8fa528a725fda67456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" is_pre_election: true
I20260924 20:18:00.922408 13038 raft_consensus.cc:2454] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cdc378b11b14b8fa528a725fda67456 in term 0.
I20260924 20:18:00.922852 13242 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a; no voters:
I20260924 20:18:00.923185 13372 raft_consensus.cc:2781] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:00.923285 13372 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:00.923331 13372 raft_consensus.cc:3037] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:00.924331 13372 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.924683 13372 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [CANDIDATE]: Term 1 election: Requested vote from peers ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705), 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991)
I20260924 20:18:00.925028 13176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c0581e22a84764b8f837ea0e8c07dd" candidate_uuid: "5cdc378b11b14b8fa528a725fda67456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef713caaba4640bcae0267dbea65a49a"
I20260924 20:18:00.925077 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c0581e22a84764b8f837ea0e8c07dd" candidate_uuid: "5cdc378b11b14b8fa528a725fda67456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
I20260924 20:18:00.925189 13038 raft_consensus.cc:3037] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:00.925165 13176 raft_consensus.cc:3037] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:00.926479 13176 raft_consensus.cc:2454] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cdc378b11b14b8fa528a725fda67456 in term 1.
I20260924 20:18:00.926487 13038 raft_consensus.cc:2454] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cdc378b11b14b8fa528a725fda67456 in term 1.
I20260924 20:18:00.926791 13243 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cdc378b11b14b8fa528a725fda67456, 971ceeba2f3c46dfb4a09a662f79ba69; no voters:
I20260924 20:18:00.926970 13372 raft_consensus.cc:2781] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:00.927208 13372 raft_consensus.cc:686] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 1 LEADER]: Becoming Leader. State: Replica: 5cdc378b11b14b8fa528a725fda67456, State: Running, Role: LEADER
I20260924 20:18:00.927579 13372 consensus_queue.cc:237] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:00.930742 12895 catalog_manager.cc:5495] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131). New cstate: current_term: 1 leader_uuid: "5cdc378b11b14b8fa528a725fda67456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:00.963014 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:00.959363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.29.83.131:41289)
0924 20:18:00.959512 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:00.959517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:00.959531 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:00.959698 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:00.959701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:00.959757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:00.959874 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:00.960110 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.960632 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.961485 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:00.961706 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:00.961877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:00.962198 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:00.962594 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:00.962718 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:00.962782 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 20:18:00.962838 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:18:00.969504 13038 raft_consensus.cc:1261] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Refusing update from remote peer 5cdc378b11b14b8fa528a725fda67456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:00.969511 13176 raft_consensus.cc:1261] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Refusing update from remote peer 5cdc378b11b14b8fa528a725fda67456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:00.970256 13375 consensus_queue.cc:1035] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:00.970434 13372 consensus_queue.cc:1035] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:00.977406 13378 mvcc.cc:205] Tried to move back new op lower bound from 7332991307642982400 to 7332991307482083328. Current Snapshot: MvccSnapshot[applied={T|T < 7332991307642982400}]
I20260924 20:18:00.979642 13380 mvcc.cc:205] Tried to move back new op lower bound from 7332991307642982400 to 7332991307482083328. Current Snapshot: MvccSnapshot[applied={T|T < 7332991307642982400}]
I20260924 20:18:00.988379 12895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:00.988545 12895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:00.991009 12895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Sending DeleteTablet for 3 replicas of tablet e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:00.991729 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:00 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:54330
I20260924 20:18:00.991804 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:00 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:41928
I20260924 20:18:00.991793 13156 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c0581e22a84764b8f837ea0e8c07dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:00 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:45596
I20260924 20:18:00.992280 13390 tablet_replica.cc:330] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: stopping tablet replica
I20260924 20:18:00.992445 13389 tablet_replica.cc:330] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: stopping tablet replica
I20260924 20:18:00.992481 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12934
I20260924 20:18:00.992830 13389 raft_consensus.cc:2229] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:00.993197 13390 raft_consensus.cc:2229] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:00.993516 13389 raft_consensus.cc:2258] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:00.993570 13390 raft_consensus.cc:2258] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:00.995343 13389 ts_tablet_manager.cc:1905] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:00.995343 13390 ts_tablet_manager.cc:1905] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:00.998160 13389 ts_tablet_manager.cc:1918] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:00.998328 13389 log.cc:1196] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:00.998733 13389 ts_tablet_manager.cc:1939] T e2c0581e22a84764b8f837ea0e8c07dd P 5cdc378b11b14b8fa528a725fda67456: Deleting consensus metadata
I20260924 20:18:00.998837 13390 ts_tablet_manager.cc:1918] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:00.998942 13390 log.cc:1196] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:00.999262 13390 ts_tablet_manager.cc:1939] T e2c0581e22a84764b8f837ea0e8c07dd P ef713caaba4640bcae0267dbea65a49a: Deleting consensus metadata
I20260924 20:18:01.000221 12883 catalog_manager.cc:4841] TS 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): tablet e2c0581e22a84764b8f837ea0e8c07dd (table test-table [id=d7c0c606ef154840a01ba17c4d99aed8]) successfully deleted
I20260924 20:18:01.000483 12881 catalog_manager.cc:4841] TS ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): tablet e2c0581e22a84764b8f837ea0e8c07dd (table test-table [id=d7c0c606ef154840a01ba17c4d99aed8]) successfully deleted
W20260924 20:18:01.001391 12882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:45991 (error 108)
I20260924 20:18:01.001564 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13088
I20260924 20:18:01.007320 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13225
I20260924 20:18:01.008915 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:01.008507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45991 (local address 127.0.0.1:41932)
0924 20:18:01.008723 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:01.008806 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260924 20:18:01.009106 12882 catalog_manager.cc:4584] TS 971ceeba2f3c46dfb4a09a662f79ba69 (127.29.83.129:45991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2c0581e22a84764b8f837ea0e8c07dd: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45991: connect: Connection refused (error 111)
I20260924 20:18:01.012454 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 12863
I20260924 20:18:21.019411 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--webserver_interface=127.29.83.190
--webserver_port=34719
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:21.131683 13392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:21.131989 13392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:21.132033 13392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:21.136294 13392 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:18:21.136358 13392 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:18:21.136373 13392 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:18:21.136387 13392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:21.136407 13392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:21.136425 13392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:21.136485 13392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:21.136507 13392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:21.142076 13392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:21.143222 13392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:21.144310 13392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:21.153652 13400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.153632 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:21.153857 13392 server_base.cc:1034] running on GCE node
W20260924 20:18:21.153612 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:21.154515 13392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:21.155658 13392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:21.156826 13392 hybrid_clock.cc:648] HybridClock initialized: now 1790281101156799 us; error 45 us; skew 500 ppm
Sep 24 20:18:21 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:21.159688 13392 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:18:21.160542 13392 webserver.cc:460] Webserver started at http://127.29.83.190:34719/ using document root <none> and password file <none>
I20260924 20:18:21.161065 13392 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:21.161125 13392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:21.164920 13392 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:21.167233 13407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.168627 13392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:21.168784 13392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5e9bb2789d57410c9d1831638f6b569a"
format_stamp: "Formatted at 2026-09-24 20:17:38 on dist-test-slave-52fr"
server_key: "6aa52da0019bc7495d67e9493d5059ab"
server_key_iv: "54bdfeccf1253933fd092558e78a4a5c"
server_key_version: "encryptionkey@0"
I20260924 20:18:21.169358 13392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:21.202354 13392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:21.206389 13392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:21.206625 13392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:21.214219 13392 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36969
I20260924 20:18:21.214519 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36969 every 8 connection(s)
I20260924 20:18:21.215377 13392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:21.216622 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13392
I20260924 20:18:21.217135 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:45991
--local_ip_for_outbound_sockets=127.29.83.129
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=44367
--webserver_interface=127.29.83.129
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:18:21.220698 13460 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:18:21.223424 13460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap starting.
I20260924 20:18:21.247485 13460 log.cc:824] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:21.261864 13460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:18:21.262382 13460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Bootstrap complete.
I20260924 20:18:21.267561 13460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:18:21.267997 13460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Initialized, Role: FOLLOWER
I20260924 20:18:21.268574 13460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:18:21.268724 13460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:21.268814 13460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:21.268934 13460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:18:21.271054 13460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:18:21.271407 13460 leader_election.cc:302] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e9bb2789d57410c9d1831638f6b569a; no voters:
I20260924 20:18:21.271736 13460 leader_election.cc:288] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:18:21.271940 13464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:18:21.272260 13464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [term 3 LEADER]: Becoming Leader. State: Replica: 5e9bb2789d57410c9d1831638f6b569a, State: Running, Role: LEADER
I20260924 20:18:21.272653 13464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } }
I20260924 20:18:21.273325 13460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:21.274113 13466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e9bb2789d57410c9d1831638f6b569a. Latest consensus state: current_term: 3 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:18:21.274231 13466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:21.274526 13465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5e9bb2789d57410c9d1831638f6b569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9bb2789d57410c9d1831638f6b569a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36969 } } }
I20260924 20:18:21.274595 13465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:21.275533 13469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:21.281316 13469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:21.282034 13469 catalog_manager.cc:1259] Loaded cluster ID: a083567d45cb459081608937fd7275f2
I20260924 20:18:21.282097 13469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:21.284487 13469 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:21.285189 13469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Loaded TSK: 0
I20260924 20:18:21.285274 13469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e9bb2789d57410c9d1831638f6b569a: Loaded TSK: 1
F20260924 20:18:21.286013 13469 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4594c7a9ad google::LogMessage::Fail() at ??:0
@ 0x7f4594c7d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4594c7a4de google::LogMessage::Flush() at ??:0
@ 0x7f4594c7ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f459793e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4597936f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f459f5e37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f459f5bd99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f459f5be178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f459f5ffdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f459f61e82b std::function<>::operator()() at ??:0
@ 0x7f459f5bdb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f459f5bef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f459f5bb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f459f5ff640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4596ecc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4596eccc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4596ece3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4596ebd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f45985016db start_thread at ??:0
@ 0x7f459377861f clone at ??:0
W20260924 20:18:21.392417 13462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:21.392690 13462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:21.392735 13462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:21.396924 13462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:21.396982 13462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:21.397076 13462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:21.402669 13462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:45991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=44367
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:21.403769 13462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:21.404826 13462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:21.414535 13507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.414577 13508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.414528 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:21.414796 13462 server_base.cc:1034] running on GCE node
I20260924 20:18:21.415448 13462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:21.416302 13462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:21.417514 13462 hybrid_clock.cc:648] HybridClock initialized: now 1790281101417453 us; error 66 us; skew 500 ppm
Sep 24 20:18:21 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:21.424942 13462 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:18:21.426182 13462 webserver.cc:460] Webserver started at http://127.29.83.129:44367/ using document root <none> and password file <none>
I20260924 20:18:21.426956 13462 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:21.427057 13462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:21.432387 13462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:18:21.435359 13517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.437062 13462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:21.437196 13462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "971ceeba2f3c46dfb4a09a662f79ba69"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "cb4a8018fc1db9aa77739bc9b02c5db6"
server_key_iv: "1c07b7df188ab81fa901ac4708411386"
server_key_version: "encryptionkey@0"
I20260924 20:18:21.437996 13462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:21.452232 13462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:21.456259 13462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:21.456468 13462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:21.457221 13462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:21.458707 13524 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:18:21.463153 13462 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:18:21.463249 13462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 20:18:21.463330 13462 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:18:21.465724 13462 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:18:21.465817 13462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:21.466224 13524 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap starting.
I20260924 20:18:21.477509 13462 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45991
I20260924 20:18:21.477829 13631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45991 every 8 connection(s)
I20260924 20:18:21.478943 13462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:21.486023 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13462
I20260924 20:18:21.486449 13524 log.cc:824] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:21.486529 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:42705
--local_ip_for_outbound_sockets=127.29.83.130
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=33567
--webserver_interface=127.29.83.130
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:18:21.496250 13524 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:18:21.496821 13524 tablet_bootstrap.cc:492] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Bootstrap complete.
I20260924 20:18:21.498400 13524 ts_tablet_manager.cc:1397] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.011s
I20260924 20:18:21.501992 13524 raft_consensus.cc:348] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:21.502561 13524 raft_consensus.cc:729] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ceeba2f3c46dfb4a09a662f79ba69, State: Initialized, Role: FOLLOWER
I20260924 20:18:21.503113 13524 consensus_queue.cc:260] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:21.504204 13524 ts_tablet_manager.cc:1428] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260924 20:18:21.526211 13634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:18:21.481452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:46669)
0924 20:18:21.482087 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.482149 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:21.483400 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:21.525173 (+ 41773us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":215,"threads_started":1}
W20260924 20:18:21.526870 13632 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36969 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:36969: Client connection negotiation failed: client connection to 127.29.83.190:36969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:18:21.527674 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.527347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:40509)
0924 20:18:21.527534 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.527608 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 20:18:21.528429 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.528180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:39871)
0924 20:18:21.528302 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.528353 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:18:21.528641 13632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:18:21.605924 13636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:21.606207 13636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:21.606292 13636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:21.610383 13636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:21.610441 13636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:21.610553 13636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:21.615967 13636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:42705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=33567
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:21.616961 13636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:21.618070 13636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:21.627787 13642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.627799 13643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.627799 13645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:21.628438 13636 server_base.cc:1034] running on GCE node
I20260924 20:18:21.628892 13636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:21.629627 13636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:21.630841 13636 hybrid_clock.cc:648] HybridClock initialized: now 1790281101630810 us; error 43 us; skew 500 ppm
Sep 24 20:18:21 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:21.633931 13636 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:18:21.634964 13636 webserver.cc:460] Webserver started at http://127.29.83.130:33567/ using document root <none> and password file <none>
I20260924 20:18:21.635576 13636 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:21.635671 13636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:21.639746 13636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:18:21.642047 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.643456 13636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:18:21.643602 13636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "ef713caaba4640bcae0267dbea65a49a"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "f36c3d794915c6fd4be677c573efe0be"
server_key_iv: "3686dcab2f1724365569f3a0a0a21a94"
server_key_version: "encryptionkey@0"
I20260924 20:18:21.644181 13636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:21.656432 13636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:21.659448 13636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:21.659648 13636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:21.660241 13636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:21.661064 13636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:21.661110 13636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.661175 13636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:21.661208 13636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.670837 13636 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:42705
I20260924 20:18:21.670854 13765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:42705 every 8 connection(s)
I20260924 20:18:21.671912 13636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:21.675048 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13636
I20260924 20:18:21.675463 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:41289
--local_ip_for_outbound_sockets=127.29.83.131
--tserver_master_addrs=127.29.83.190:36969
--webserver_port=37425
--webserver_interface=127.29.83.131
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:18:21.676116 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.674155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:44525)
0924 20:18:21.674652 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.674767 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
W20260924 20:18:21.676750 13766 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36969 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:36969: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
I20260924 20:18:21.677901 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.677413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:38463)
0924 20:18:21.677548 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.677613 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:18:21.678624 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.678372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:44017)
0924 20:18:21.678508 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.678561 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 20:18:21.678813 13766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:18:21.791589 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:21.791915 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:21.792002 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:21.796211 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:21.796312 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:21.796409 13769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:21.801719 13769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:41289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=37425
--tserver_master_addrs=127.29.83.190:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:21.802907 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:21.803983 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:21.814026 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.814011 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:21.814015 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:21.814280 13769 server_base.cc:1034] running on GCE node
I20260924 20:18:21.814828 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:21.815533 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:21.816756 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790281101816724 us; error 44 us; skew 500 ppm
Sep 24 20:18:21 dist-test-slave-52fr krb5kdc[12314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:21.819950 13769 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:18:21.820927 13769 webserver.cc:460] Webserver started at http://127.29.83.131:37425/ using document root <none> and password file <none>
I20260924 20:18:21.821550 13769 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:21.821650 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:21.825660 13769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:21.827924 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.829303 13769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:21.829427 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "5cdc378b11b14b8fa528a725fda67456"
format_stamp: "Formatted at 2026-09-24 20:17:39 on dist-test-slave-52fr"
server_key: "3aeabb3ff5bcf353874b0294af19c1f6"
server_key_iv: "754c4d0102600adb35f37d325eb99ece"
server_key_version: "encryptionkey@0"
I20260924 20:18:21.830067 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:21.848093 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:21.851729 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:21.851890 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:21.852545 13769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:21.853353 13769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:21.853398 13769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.853437 13769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:21.853453 13769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:21.862778 13769 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:41289
I20260924 20:18:21.862800 13897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:41289 every 8 connection(s)
I20260924 20:18:21.863847 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:21.868160 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.866208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:48791)
0924 20:18:21.866762 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.866892 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
W20260924 20:18:21.868782 13898 heartbeater.cc:644] Failed to heartbeat to 127.29.83.190:36969 (0 consecutive failures): Network error: Failed to ping master at 127.29.83.190:36969: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
I20260924 20:18:21.869576 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.869305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:33123)
0924 20:18:21.869441 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.869503 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 20:18:21.870306 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:21.870084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:40105)
0924 20:18:21.870198 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:21.870252 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:18:21.870538 13898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:18:21.872716 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13769
I20260924 20:18:22.530061 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:22.529333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:43539)
0924 20:18:22.529684 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:22.529801 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
I20260924 20:18:22.679876 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:22.679438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:42403)
0924 20:18:22.679711 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:22.679772 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260924 20:18:22.871546 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:22.871111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:37837)
0924 20:18:22.871379 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:22.871443 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260924 20:18:23.251812 13905 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:23.252048 13905 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:23.253194 13905 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
Sep 24 20:18:23 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.131@KRBTEST.COM
Sep 24 20:18:23 dist-test-slave-52fr krb5kdc[12314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.130@KRBTEST.COM
I20260924 20:18:23.269152 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.253955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:56403 (local address 127.29.83.131:41289)
0924 20:18:23.254266 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:23.254291 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:23.254334 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:18:23.254982 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:23.254989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:23.255339 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:23.255660 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:23.256978 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.258087 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.259662 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.260380 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.260578 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:23.264304 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:23.264309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:23.264324 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:23.264357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:23.266399 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267021 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:23.267026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:23.267031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:23.267153 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267505 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:23.267510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:23.267511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:23.267951 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:23.268112 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:23.268674 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 20:18:23.268944 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260924 20:18:23.269153 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.253449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:41289 (local address 127.29.83.129:56403)
0924 20:18:23.253852 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:23.253869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:23.253992 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:23.255738 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:23.255744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:23.256286 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:23.256783 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:23.256798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.258242 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.258246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:23.259507 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:23.259519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.260562 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.260567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:23.260859 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:23.261812 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:23.261840 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:23.264027 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:23.266548 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:23.266554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:23.266569 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:23.266876 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:23.267253 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:23.267260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:23.267406 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:23.268089 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:23.268099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:23.268368 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 20:18:23.268565 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 20:18:23.268956 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":336,"spinlock_wait_cycles":33664,"thread_start_us":82,"threads_started":1}
I20260924 20:18:23.269169 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.253725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34513 (local address 127.29.83.130:42705)
0924 20:18:23.254117 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:23.254133 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:23.254249 (+ 116us) server_negotiation.cc:407] Connection header received
0924 20:18:23.254983 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:23.254989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:23.255339 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:23.255704 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:23.256981 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.258219 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.259662 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.260380 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.260578 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:23.264582 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:23.264586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:23.264600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:23.264639 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:23.266500 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267072 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:23.267084 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:23.267090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:23.267208 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267518 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:23.267525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:23.267527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:23.267951 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:23.268114 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:23.268691 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 20:18:23.268976 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":83,"threads_started":1}
I20260924 20:18:23.269866 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.253461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:42705 (local address 127.29.83.129:34513)
0924 20:18:23.254133 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:23.254146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:23.254245 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:23.255738 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:23.255744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:23.256383 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:23.256785 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:23.256799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.258356 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.258360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:23.259507 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:23.259519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:23.260562 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:23.260567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:23.260849 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:23.261812 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:23.261853 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:23.264351 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:23.266635 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:23.266640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:23.266643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:23.266947 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:23.267309 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:23.267313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:23.267315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:23.267412 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:23.268089 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:23.268096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:23.268369 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 20:18:23.268566 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 20:18:23.269052 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":450,"spinlock_wait_cycles":44672,"thread_start_us":74,"threads_started":1}
W20260924 20:18:23.271889 13521 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
W20260924 20:18:23.272172 13519 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:23.272287 13519 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69; no voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a
I20260924 20:18:23.272658 13905 raft_consensus.cc:2726] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:18:23.531369 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.531009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:51115)
0924 20:18:23.531189 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:23.531269 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 20:18:23.681305 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.680739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:52475)
0924 20:18:23.681100 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:23.681181 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260924 20:18:23.872773 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:23.872332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:42087)
0924 20:18:23.872609 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:23.872680 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260924 20:18:24.532861 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:24.532414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:32999)
0924 20:18:24.532696 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:24.532757 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260924 20:18:24.682695 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:24.682245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:48625)
0924 20:18:24.682533 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:24.682597 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260924 20:18:24.874298 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:24.873750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:38621)
0924 20:18:24.874133 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:24.874200 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260924 20:18:25.165138 13915 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:25.165284 13915 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:25.165802 13915 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
W20260924 20:18:25.166711 13519 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
W20260924 20:18:25.166832 13521 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:25.166937 13521 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69; no voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a
I20260924 20:18:25.167141 13915 raft_consensus.cc:2726] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:18:25.534267 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:25.533734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:33251)
0924 20:18:25.534094 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:25.534170 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":88,"threads_started":1}
I20260924 20:18:25.683832 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:25.683419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:49747)
0924 20:18:25.683681 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:25.683737 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260924 20:18:25.875727 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:25.875293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:36711)
0924 20:18:25.875574 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:25.875631 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260924 20:18:26.535913 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:26.535455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:44685)
0924 20:18:26.535747 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:26.535816 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260924 20:18:26.685280 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:26.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:40261)
0924 20:18:26.685097 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:26.685176 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260924 20:18:26.877040 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:26.876619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:57407)
0924 20:18:26.876885 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:26.876946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260924 20:18:27.537153 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:27.536703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:34305)
0924 20:18:27.536992 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:27.537056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260924 20:18:27.686770 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:27.686273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:33363)
0924 20:18:27.686591 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:27.686665 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260924 20:18:27.878438 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:27.877968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:45651)
0924 20:18:27.878247 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:27.878309 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260924 20:18:28.451337 13925 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:28.451480 13925 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:28.451964 13925 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
W20260924 20:18:28.452821 13519 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
W20260924 20:18:28.452975 13521 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:28.453076 13521 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69; no voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a
I20260924 20:18:28.453223 13925 raft_consensus.cc:2726] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:18:28.538388 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:28.537966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:37223)
0924 20:18:28.538233 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:28.538290 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260924 20:18:28.688138 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:28.687601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:50369)
0924 20:18:28.687946 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:28.688031 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
I20260924 20:18:28.879721 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:28.879277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:39135)
0924 20:18:28.879571 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:28.879625 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
I20260924 20:18:29.539685 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:29.539270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:48941)
0924 20:18:29.539525 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:29.539588 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260924 20:18:29.689366 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:29.688911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:55711)
0924 20:18:29.689210 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:29.689269 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":121,"threads_started":1}
I20260924 20:18:29.881026 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:29.880558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:48339)
0924 20:18:29.880862 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:29.880927 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260924 20:18:30.540931 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:30.540500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:54967)
0924 20:18:30.540777 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:30.540834 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260924 20:18:30.690763 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:30.690310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:34629)
0924 20:18:30.690595 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:30.690657 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260924 20:18:30.882265 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:30.881860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:60661)
0924 20:18:30.882103 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:30.882166 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260924 20:18:31.458671 13935 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:31.458838 13935 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:31.459270 13935 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
W20260924 20:18:31.460203 13521 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
W20260924 20:18:31.460302 13519 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:31.460433 13519 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69; no voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a
I20260924 20:18:31.460587 13935 raft_consensus.cc:2726] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:18:31.542478 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:31.541984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:37201)
0924 20:18:31.542295 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:31.542375 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260924 20:18:31.692078 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:31.691620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:41205)
0924 20:18:31.691900 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:31.691966 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260924 20:18:31.883598 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:31.883162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:53559)
0924 20:18:31.883433 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:31.883499 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260924 20:18:32.543790 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:32.543343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:47037)
0924 20:18:32.543627 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:32.543692 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260924 20:18:32.693238 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:32.692821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:58045)
0924 20:18:32.693075 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:32.693129 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260924 20:18:32.884970 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:32.884519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:47133)
0924 20:18:32.884800 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:32.884862 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260924 20:18:33.545158 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:33.544715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:39001)
0924 20:18:33.544994 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:33.545061 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260924 20:18:33.694540 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:33.694089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:55505)
0924 20:18:33.694387 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:33.694444 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260924 20:18:33.886523 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:33.886078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:57677)
0924 20:18:33.886333 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:33.886423 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260924 20:18:34.546593 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:34.546089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:57435)
0924 20:18:34.546426 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:34.546492 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260924 20:18:34.695734 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:34.695328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:46053)
0924 20:18:34.695574 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:34.695636 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260924 20:18:34.887838 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:34.887438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:38183)
0924 20:18:34.887689 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:34.887743 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260924 20:18:35.548115 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:35.547621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:37275)
0924 20:18:35.547936 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:35.548015 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260924 20:18:35.697010 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:35.696560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:34171)
0924 20:18:35.696837 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:35.696909 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260924 20:18:35.889151 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:35.888721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.131:55239)
0924 20:18:35.888993 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:35.889055 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260924 20:18:36.549641 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:36.549149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.129:60133)
0924 20:18:36.549471 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:36.549541 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260924 20:18:36.646044 13952 raft_consensus.cc:482] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:36.646200 13952 raft_consensus.cc:504] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cdc378b11b14b8fa528a725fda67456" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 41289 } } peers { permanent_uuid: "ef713caaba4640bcae0267dbea65a49a" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 42705 } } peers { permanent_uuid: "971ceeba2f3c46dfb4a09a662f79ba69" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45991 } }
I20260924 20:18:36.646710 13952 leader_election.cc:288] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289), ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705)
W20260924 20:18:36.647567 13521 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cdc378b11b14b8fa528a725fda67456 (127.29.83.131:41289): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
W20260924 20:18:36.647677 13519 leader_election.cc:341] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef713caaba4640bcae0267dbea65a49a (127.29.83.130:42705): Not found: Tablet not found: e2c0581e22a84764b8f837ea0e8c07dd
I20260924 20:18:36.647830 13519 leader_election.cc:302] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971ceeba2f3c46dfb4a09a662f79ba69; no voters: 5cdc378b11b14b8fa528a725fda67456, ef713caaba4640bcae0267dbea65a49a
I20260924 20:18:36.647974 13952 raft_consensus.cc:2726] T e2c0581e22a84764b8f837ea0e8c07dd P 971ceeba2f3c46dfb4a09a662f79ba69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:18:36.698642 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:36.698137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:36969 (local address 127.29.83.130:55071)
0924 20:18:36.698462 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:36.698530 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.190:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260924 20:18:36.877415 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13462
I20260924 20:18:36.883409 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13636
I20260924 20:18:36.889014 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13769
2026-09-24T20:18:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59455 ms)
[----------] 30 tests from SecurityITest (332263 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:36 dist-test-slave-52fr krb5kdc[13956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:36 dist-test-slave-52fr krb5kdc[13956](info): set up 2 sockets
Sep 24 20:18:36 dist-test-slave-52fr krb5kdc[13956](info): commencing operation
krb5kdc: starting...
W20260924 20:18:37.966207 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:18:38.097015 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45735
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:38.209257 13969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:38.209573 13969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:38.209617 13969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:38.214555 13969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:38.214625 13969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:38.214646 13969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:38.214663 13969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:38.214679 13969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:38.220276 13969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:38.221407 13969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:38.222503 13969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:38.231187 13977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.231173 13975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.231173 13974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:38.231734 13969 server_base.cc:1034] running on GCE node
I20260924 20:18:38.232347 13969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:38.233580 13969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:38.234782 13969 hybrid_clock.cc:648] HybridClock initialized: now 1790281118234762 us; error 43 us; skew 500 ppm
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:38.237926 13969 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:18:38.238933 13969 webserver.cc:460] Webserver started at http://127.29.83.190:38857/ using document root <none> and password file <none>
I20260924 20:18:38.239511 13969 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:38.239575 13969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:38.239866 13969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:38.242274 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "68aabe64e2794c2a97b5e145edcc9fd3"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0a1a6713fba99f52dd6f708104e92d74"
server_key_iv: "c40f2a5f1364ede67d3dda9e538d32a5"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.242959 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "68aabe64e2794c2a97b5e145edcc9fd3"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0a1a6713fba99f52dd6f708104e92d74"
server_key_iv: "c40f2a5f1364ede67d3dda9e538d32a5"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.247063 13969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 20:18:38.249686 13984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.251080 13969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:38.251215 13969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "68aabe64e2794c2a97b5e145edcc9fd3"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0a1a6713fba99f52dd6f708104e92d74"
server_key_iv: "c40f2a5f1364ede67d3dda9e538d32a5"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.251355 13969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:38.291617 13969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:38.295274 13969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:38.295487 13969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:38.303181 13969 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:45735
I20260924 20:18:38.303202 14036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:45735 every 8 connection(s)
I20260924 20:18:38.304175 13969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:38.307077 14037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:38.313115 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: Bootstrap starting.
I20260924 20:18:38.314220 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 13969
I20260924 20:18:38.314376 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:38.314708 30030 external_mini_cluster.cc:1428] Setting key 20304d39d183b578f7455aab2ec3075e
I20260924 20:18:38.316394 14037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:38.317167 14037 log.cc:824] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:38.319612 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: No bootstrap required, opened a new log
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:38.324435 14037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } }
I20260924 20:18:38.324740 14037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:38.324774 14037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aabe64e2794c2a97b5e145edcc9fd3, State: Initialized, Role: FOLLOWER
I20260924 20:18:38.325376 14037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } }
I20260924 20:18:38.325554 14037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:38.325654 14037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:38.325799 14037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:38.326987 14037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } }
I20260924 20:18:38.327355 14037 leader_election.cc:302] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68aabe64e2794c2a97b5e145edcc9fd3; no voters:
I20260924 20:18:38.327749 14037 leader_election.cc:288] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:38.327886 14042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:38.328146 14042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 68aabe64e2794c2a97b5e145edcc9fd3, State: Running, Role: LEADER
I20260924 20:18:38.328529 14042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } }
I20260924 20:18:38.329172 14037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:38.331050 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:38.315907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.29.83.190:45735)
0924 20:18:38.316355 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:38.316370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:38.316463 (+ 93us) server_negotiation.cc:407] Connection header received
0924 20:18:38.317181 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:38.317199 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:38.317660 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:38.318082 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:38.318997 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.320080 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.321330 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.322050 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.322198 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:38.325439 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:38.325445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:38.325459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:38.325492 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:38.327443 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.328086 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.328095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.328106 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.328272 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.328602 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.328606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.328608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.329130 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:38.329314 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:38.329673 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 20:18:38.330029 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260924 20:18:38.332058 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } } }
I20260924 20:18:38.332270 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:38.332842 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68aabe64e2794c2a97b5e145edcc9fd3. Latest consensus state: current_term: 1 leader_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68aabe64e2794c2a97b5e145edcc9fd3" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45735 } } }
I20260924 20:18:38.332959 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3 [sys.catalog]: This master's current role is: LEADER
W20260924 20:18:38.335041 14057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:18:38.335124 14057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:18:38.335199 14058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:38.336087 14058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:38.341683 14058 catalog_manager.cc:1347] Generated new cluster ID: 23a545f798654dc8b97076a382cebf19
I20260924 20:18:38.341789 14058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:38.349841 14058 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:38.350886 14058 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:38.365479 14058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: Generated new TSK 0
I20260924 20:18:38.366330 14058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:18:38.402274 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:38.529793 14063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:38.530134 14063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:38.530231 14063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:38.534716 14063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:38.534808 14063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:38.534902 14063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:38.541193 14063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:38.542814 14063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:38.544032 14063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:18:38.555691 14063 server_base.cc:1034] running on GCE node
W20260924 20:18:38.555630 14069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.555614 14071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.555653 14068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:38.556406 14063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:38.557212 14063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:38.558416 14063 hybrid_clock.cc:648] HybridClock initialized: now 1790281118558395 us; error 36 us; skew 500 ppm
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:38.562043 14063 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:18:38.563055 14063 webserver.cc:460] Webserver started at http://127.29.83.129:36277/ using document root <none> and password file <none>
I20260924 20:18:38.563664 14063 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:38.563730 14063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:38.563964 14063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:38.566197 14063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "29142a4fd630400d9c0d022689a38fd1"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "2ff097a5dc693f0c358556263544fbab"
server_key_iv: "91881ba756df891a7272ccdcaf9e875f"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.566795 14063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "29142a4fd630400d9c0d022689a38fd1"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "2ff097a5dc693f0c358556263544fbab"
server_key_iv: "91881ba756df891a7272ccdcaf9e875f"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.570909 14063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:38.573417 14078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.574755 14063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:18:38.574909 14063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "29142a4fd630400d9c0d022689a38fd1"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "2ff097a5dc693f0c358556263544fbab"
server_key_iv: "91881ba756df891a7272ccdcaf9e875f"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.575031 14063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:38.595335 14063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:38.598994 14063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:38.599215 14063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:38.599846 14063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:38.600716 14063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:38.600788 14063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.600884 14063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:38.600930 14063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.610339 14063 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:38503
I20260924 20:18:38.610391 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:38503 every 8 connection(s)
I20260924 20:18:38.611291 14063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:38.620692 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14063
I20260924 20:18:38.620846 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:38.621196 30030 external_mini_cluster.cc:1428] Setting key 05dabd8ff64315261faf7c0c1f6ed181
I20260924 20:18:38.625408 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:38.613168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:34879 (local address 127.29.83.190:45735)
0924 20:18:38.613347 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:38.613351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:38.614260 (+ 909us) server_negotiation.cc:407] Connection header received
0924 20:18:38.615553 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:38.615556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:38.615604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:38.615682 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:38.616855 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.617424 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.618692 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.618965 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.619071 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:38.621672 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:38.621675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:38.621677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:38.621701 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:38.623290 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.623821 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.623825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.623827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.623922 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.624258 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.624263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.624265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.624462 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:38.624565 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:38.625102 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 20:18:38.625234 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 20:18:38.626363 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:38.613592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45735 (local address 127.29.83.129:34879)
0924 20:18:38.614105 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:38.614149 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:38.615324 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:38.615820 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:38.615827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:38.616195 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:38.616705 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:38.616720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.617567 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.617571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:38.618557 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:38.618567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.619125 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.619128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:38.619270 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:38.619842 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:38.619864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:38.621481 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:38.623435 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:38.623441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:38.623462 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:38.623715 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:38.624013 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:38.624018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:38.624020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:38.624160 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:38.624576 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:38.624585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:38.624851 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 20:18:38.625063 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 20:18:38.625412 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":167,"threads_started":1}
I20260924 20:18:38.627740 14192 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45735
I20260924 20:18:38.628053 14192 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:38.628717 14192 heartbeater.cc:511] Master 127.29.83.190:45735 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:18:38.630760 14001 ts_manager.cc:194] Registered new tserver with Master: 29142a4fd630400d9c0d022689a38fd1 (127.29.83.129:38503)
I20260924 20:18:38.631950 14001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:34879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:18:38.652745 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:38.767815 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:38.768190 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:38.768261 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:38.773137 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:38.773250 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:38.773376 14197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:38.779886 14197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:38.781033 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:38.782208 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:38.792249 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.792240 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:38.792240 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:38.792335 14197 server_base.cc:1034] running on GCE node
I20260924 20:18:38.792943 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:38.793641 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:38.794869 14197 hybrid_clock.cc:648] HybridClock initialized: now 1790281118794833 us; error 31 us; skew 500 ppm
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:38.797672 14197 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:18:38.798683 14197 webserver.cc:460] Webserver started at http://127.29.83.130:39177/ using document root <none> and password file <none>
I20260924 20:18:38.799290 14197 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:38.799387 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:38.799623 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:38.801712 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "f6027b6d2046425cad68647ce6c49497"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0abd1ff1649413acace7201afe89fa91"
server_key_iv: "9cc2dd7a19efef3f37b8bd2749aaa2ab"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.802366 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "f6027b6d2046425cad68647ce6c49497"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0abd1ff1649413acace7201afe89fa91"
server_key_iv: "9cc2dd7a19efef3f37b8bd2749aaa2ab"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.806273 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:38.808835 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.810271 14197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:38.810422 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "f6027b6d2046425cad68647ce6c49497"
format_stamp: "Formatted at 2026-09-24 20:18:38 on dist-test-slave-52fr"
server_key: "0abd1ff1649413acace7201afe89fa91"
server_key_iv: "9cc2dd7a19efef3f37b8bd2749aaa2ab"
server_key_version: "encryptionkey@0"
I20260924 20:18:38.810541 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:38.823220 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:38.826887 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:38.827107 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:38.827770 14197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:38.828661 14197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:38.828732 14197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.828830 14197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:38.828862 14197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:38.839246 14197 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:39623
I20260924 20:18:38.839359 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:39623 every 8 connection(s)
I20260924 20:18:38.840296 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:38.850113 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14197
I20260924 20:18:38.850265 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:38.850622 30030 external_mini_cluster.cc:1428] Setting key 209735db4ebe398686cd0a30d4a3d0bb
Sep 24 20:18:38 dist-test-slave-52fr krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:38.856426 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:38.842407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:58035 (local address 127.29.83.190:45735)
0924 20:18:38.842609 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:38.842613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:38.843500 (+ 887us) server_negotiation.cc:407] Connection header received
0924 20:18:38.844950 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:38.844954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:38.845013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:38.845126 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:38.846437 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.846980 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.848189 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.848421 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.848539 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:38.851642 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:38.851647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:38.851649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:38.851674 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:38.853444 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.854209 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.854216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.854220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.854356 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:38.854800 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:38.854806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:38.854809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:38.855084 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:38.855187 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:38.856025 (+ 838us) server_negotiation.cc:299] Negotiation successful
0924 20:18:38.856176 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 20:18:38.857342 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:38.842813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45735 (local address 127.29.83.130:58035)
0924 20:18:38.843312 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:38.843369 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:38.844651 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:38.845315 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:38.845322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:38.845699 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:38.846276 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:38.846301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.847124 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.847127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:38.848059 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:38.848069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:38.848586 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:38.848590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:38.848770 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:38.849433 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:38.849450 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:38.851394 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:38.853583 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:38.853594 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:38.853617 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:38.854090 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:38.854465 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:38.854468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:38.854470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:38.854691 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:38.855204 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:38.855210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:38.855729 (+ 519us) client_negotiation.cc:769] Sending connection context
0924 20:18:38.855925 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 20:18:38.856243 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
I20260924 20:18:38.858875 14326 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45735
I20260924 20:18:38.859417 14326 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:38.860193 14326 heartbeater.cc:511] Master 127.29.83.190:45735 requested a full tablet report, sending...
I20260924 20:18:38.861821 14001 ts_manager.cc:194] Registered new tserver with Master: f6027b6d2046425cad68647ce6c49497 (127.29.83.130:39623)
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:18:38.862690 14001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:58035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:18:38.886528 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:39.002585 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:39.002878 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:39.002923 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:39.007040 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:39.007118 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:39.007180 14331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:39.012669 14331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:39.013859 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:39.014961 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:39.024896 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:39.024966 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:39.024993 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:39.025349 14331 server_base.cc:1034] running on GCE node
I20260924 20:18:39.025904 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:39.026633 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:39.027863 14331 hybrid_clock.cc:648] HybridClock initialized: now 1790281119027840 us; error 35 us; skew 500 ppm
Sep 24 20:18:39 dist-test-slave-52fr krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:39.031858 14331 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:18:39.032814 14331 webserver.cc:460] Webserver started at http://127.29.83.131:38467/ using document root <none> and password file <none>
I20260924 20:18:39.033409 14331 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:39.033555 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:39.033843 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:39.035887 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "55610796e8db4437b5b56d13bbb3030c"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "089220db39d0032022c5f142a2b3fbab"
server_key_iv: "ed333a8993b5d8cc84a47ecc5c30b176"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.036466 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "55610796e8db4437b5b56d13bbb3030c"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "089220db39d0032022c5f142a2b3fbab"
server_key_iv: "ed333a8993b5d8cc84a47ecc5c30b176"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.040555 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:18:39.043239 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:39.044659 14331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:39.044770 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "55610796e8db4437b5b56d13bbb3030c"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "089220db39d0032022c5f142a2b3fbab"
server_key_iv: "ed333a8993b5d8cc84a47ecc5c30b176"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.044919 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:39.067912 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:39.071731 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:39.071980 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:39.072652 14331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:39.073541 14331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:39.073589 14331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:39.073630 14331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:39.073673 14331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:39.082897 14331 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:44449
I20260924 20:18:39.083039 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:44449 every 8 connection(s)
I20260924 20:18:39.084031 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:39.094153 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14331
I20260924 20:18:39.094297 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
Sep 24 20:18:39 dist-test-slave-52fr krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:39.094614 30030 external_mini_cluster.cc:1428] Setting key 22b80af113fa290a08efdb688899d181
I20260924 20:18:39.099308 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.086142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:58113 (local address 127.29.83.190:45735)
0924 20:18:39.086300 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.086303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.087319 (+ 1016us) server_negotiation.cc:407] Connection header received
0924 20:18:39.088699 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.088702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.088766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.088870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:39.090207 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.090759 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.091968 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.092194 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.092307 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.095219 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:39.095223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:39.095226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:39.095255 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:39.097075 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:39.097646 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:39.097649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:39.097651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:39.097739 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:39.098121 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:39.098123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:39.098125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:39.098318 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:39.098400 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.098976 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.099088 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:18:39.100138 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.086593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45735 (local address 127.29.83.131:58113)
0924 20:18:39.087148 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.087205 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.088466 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.089067 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.089074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.089450 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:39.090058 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.090073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.090907 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.090910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.091817 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.091826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.092357 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.092362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.092557 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.093200 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:39.093217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:39.095050 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:39.097221 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:39.097227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:39.097242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:39.097533 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:39.097884 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:39.097886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:39.097888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:39.098035 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:39.098445 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:39.098451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:39.098723 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.098921 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.099240 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":123,"threads_started":1}
I20260924 20:18:39.101361 14460 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45735
I20260924 20:18:39.101661 14460 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:39.102208 14460 heartbeater.cc:511] Master 127.29.83.190:45735 requested a full tablet report, sending...
I20260924 20:18:39.103327 14001 ts_manager.cc:194] Registered new tserver with Master: 55610796e8db4437b5b56d13bbb3030c (127.29.83.131:44449)
I20260924 20:18:39.103998 14001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:58113
I20260924 20:18:39.109375 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:39.119668 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.111507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.29.83.190:45735)
0924 20:18:39.111671 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.111676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.111831 (+ 155us) server_negotiation.cc:407] Connection header received
0924 20:18:39.112005 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.112008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.112059 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.112144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:39.112863 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.113387 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.114416 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.114644 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.114756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.116406 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:39.116416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:39.116418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:39.116444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:39.117926 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:39.118371 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:39.118375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:39.118377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:39.118463 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:39.118750 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:39.118754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:39.118755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:39.118988 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:39.119131 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.119377 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.119499 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:18:39.123072 14001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:39.126469 14001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:39.141340 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.136269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.29.83.131:44449)
0924 20:18:39.136750 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.136764 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.136788 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:18:39.136871 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.136876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.137065 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.137224 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:39.137371 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138301 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.139940 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.140642 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.140785 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.140946 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.141093 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.141188 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":114,"threads_started":1}
I20260924 20:18:39.141332 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.135655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:44449 (local address 127.0.0.1:58554)
0924 20:18:39.136101 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.136113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.136209 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.137201 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.137203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.137217 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:39.137290 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.137299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138448 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.138455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.139812 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.139822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.140787 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.140790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.140908 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.140944 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.141052 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.141179 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":27,"thread_start_us":98,"threads_started":1}
I20260924 20:18:39.141611 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.135323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:38503 (local address 127.0.0.1:59634)
0924 20:18:39.135734 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.135793 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.135923 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.137054 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.137066 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.137088 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:39.137258 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.137266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138857 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.138868 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.139957 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.139964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.141074 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.141081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.141177 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.141198 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.141245 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.141335 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":128,"threads_started":1}
I20260924 20:18:39.141934 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.135506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.29.83.129:38503)
0924 20:18:39.136540 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.136548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.136572 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:18:39.136648 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.136652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.136875 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.137416 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:39.137540 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138736 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.140068 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.140925 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.141430 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.141591 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.141642 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.141790 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":80,"threads_started":1}
I20260924 20:18:39.142349 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.136282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.29.83.130:39623)
0924 20:18:39.136911 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.136922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.136940 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:18:39.137001 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.137005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.137155 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.137279 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:39.137696 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138820 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.140236 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.141044 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.141135 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.141285 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.142106 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.142204 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":129,"threads_started":1}
I20260924 20:18:39.141978 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.136117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:39623 (local address 127.0.0.1:58838)
0924 20:18:39.136384 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.136397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.136474 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.137463 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.137466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.137482 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:39.137560 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.137570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.138960 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.138966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.140111 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.140122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.141342 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.141346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.141584 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.141610 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.141777 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.141843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":31,"thread_start_us":98,"threads_started":1}
I20260924 20:18:39.144737 14126 tablet_service.cc:1467] Processing CreateTablet for tablet cdfce2e723184412b535e01feaf96b26 (DEFAULT_TABLE table=test-table [id=01d8ca3b129243b5ab2faf71526729ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:39.144958 14260 tablet_service.cc:1467] Processing CreateTablet for tablet cdfce2e723184412b535e01feaf96b26 (DEFAULT_TABLE table=test-table [id=01d8ca3b129243b5ab2faf71526729ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:39.145879 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfce2e723184412b535e01feaf96b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:39.145998 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfce2e723184412b535e01feaf96b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:39.147732 14394 tablet_service.cc:1467] Processing CreateTablet for tablet cdfce2e723184412b535e01feaf96b26 (DEFAULT_TABLE table=test-table [id=01d8ca3b129243b5ab2faf71526729ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:39.148825 14394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfce2e723184412b535e01feaf96b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:39.152864 14477 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Bootstrap starting.
I20260924 20:18:39.154369 14478 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Bootstrap starting.
I20260924 20:18:39.155387 14477 tablet_bootstrap.cc:654] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:39.156262 14477 log.cc:824] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:39.156291 14479 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Bootstrap starting.
I20260924 20:18:39.156661 14478 tablet_bootstrap.cc:654] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:39.157972 14478 log.cc:824] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:39.158646 14479 tablet_bootstrap.cc:654] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:39.159063 14477 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: No bootstrap required, opened a new log
I20260924 20:18:39.159277 14477 ts_tablet_manager.cc:1397] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:18:39.159852 14479 log.cc:824] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:39.160027 14478 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: No bootstrap required, opened a new log
I20260924 20:18:39.160221 14478 ts_tablet_manager.cc:1397] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:18:39.162045 14479 tablet_bootstrap.cc:492] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: No bootstrap required, opened a new log
I20260924 20:18:39.162297 14479 ts_tablet_manager.cc:1397] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:18:39.162958 14477 raft_consensus.cc:348] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.163226 14477 raft_consensus.cc:374] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:39.163282 14477 raft_consensus.cc:729] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29142a4fd630400d9c0d022689a38fd1, State: Initialized, Role: FOLLOWER
I20260924 20:18:39.164021 14477 consensus_queue.cc:260] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.164383 14478 raft_consensus.cc:348] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.164642 14478 raft_consensus.cc:374] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:39.164690 14478 raft_consensus.cc:729] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6027b6d2046425cad68647ce6c49497, State: Initialized, Role: FOLLOWER
I20260924 20:18:39.165072 14477 ts_tablet_manager.cc:1428] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:18:39.165131 14478 consensus_queue.cc:260] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.165475 14192 heartbeater.cc:503] Master 127.29.83.190:45735 was elected leader, sending a full tablet report...
I20260924 20:18:39.165922 14326 heartbeater.cc:503] Master 127.29.83.190:45735 was elected leader, sending a full tablet report...
I20260924 20:18:39.166159 14478 ts_tablet_manager.cc:1428] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:18:39.166338 14479 raft_consensus.cc:348] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.166855 14479 raft_consensus.cc:374] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:39.166925 14479 raft_consensus.cc:729] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55610796e8db4437b5b56d13bbb3030c, State: Initialized, Role: FOLLOWER
I20260924 20:18:39.167347 14479 consensus_queue.cc:260] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.167984 14460 heartbeater.cc:503] Master 127.29.83.190:45735 was elected leader, sending a full tablet report...
I20260924 20:18:39.168304 14479 ts_tablet_manager.cc:1428] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 20:18:39.335107 14461 tablet.cc:2369] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:39.341537 14327 tablet.cc:2369] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:39.362850 14193 tablet.cc:2369] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:39.501192 14484 raft_consensus.cc:482] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:39.501395 14484 raft_consensus.cc:504] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.502638 14484 leader_election.cc:288] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29142a4fd630400d9c0d022689a38fd1 (127.29.83.129:38503), 55610796e8db4437b5b56d13bbb3030c (127.29.83.131:44449)
I20260924 20:18:39.506542 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.503021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:49959 (local address 127.29.83.131:44449)
0924 20:18:39.503137 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.503142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.503393 (+ 251us) server_negotiation.cc:407] Connection header received
0924 20:18:39.503527 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.503531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.503579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.503657 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:39.503922 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.504426 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.505494 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.505930 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.506059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.506118 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.506329 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.506394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:18:39.506546 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.502968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:42261 (local address 127.29.83.129:38503)
0924 20:18:39.503100 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.503108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.503262 (+ 154us) server_negotiation.cc:407] Connection header received
0924 20:18:39.503364 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.503368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.503418 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.503498 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:39.503805 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.504360 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.505454 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.505930 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.506054 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.506118 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.506351 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.506422 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:39.506577 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.502907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:44449 (local address 127.29.83.130:49959)
0924 20:18:39.503275 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.503290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.503388 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.503678 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.503680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.503694 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:39.503814 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.503820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.504550 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.504553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.505338 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.505346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.506051 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.506055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.506202 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.506225 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.506284 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.506373 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":30,"thread_start_us":74,"threads_started":1}
I20260924 20:18:39.506579 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.502810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:38503 (local address 127.29.83.130:42261)
0924 20:18:39.503143 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.503159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:39.503277 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:39.503523 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:39.503526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:39.503544 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:39.503690 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.503698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.504473 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.504479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.505331 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:39.505339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.506049 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.506052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:39.506210 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.506231 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:18:39.506324 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:18:39.506394 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":47,"thread_start_us":89,"threads_started":1}
I20260924 20:18:39.507212 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfce2e723184412b535e01feaf96b26" candidate_uuid: "f6027b6d2046425cad68647ce6c49497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55610796e8db4437b5b56d13bbb3030c" is_pre_election: true
I20260924 20:18:39.507320 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfce2e723184412b535e01feaf96b26" candidate_uuid: "f6027b6d2046425cad68647ce6c49497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29142a4fd630400d9c0d022689a38fd1" is_pre_election: true
I20260924 20:18:39.507543 14146 raft_consensus.cc:2454] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6027b6d2046425cad68647ce6c49497 in term 0.
I20260924 20:18:39.507539 14414 raft_consensus.cc:2454] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6027b6d2046425cad68647ce6c49497 in term 0.
I20260924 20:18:39.508037 14214 leader_election.cc:302] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29142a4fd630400d9c0d022689a38fd1, f6027b6d2046425cad68647ce6c49497; no voters:
I20260924 20:18:39.508291 14484 raft_consensus.cc:2781] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:39.508380 14484 raft_consensus.cc:482] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:39.508424 14484 raft_consensus.cc:3037] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:39.509502 14484 raft_consensus.cc:504] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.509931 14484 leader_election.cc:288] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [CANDIDATE]: Term 1 election: Requested vote from peers 29142a4fd630400d9c0d022689a38fd1 (127.29.83.129:38503), 55610796e8db4437b5b56d13bbb3030c (127.29.83.131:44449)
I20260924 20:18:39.510304 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfce2e723184412b535e01feaf96b26" candidate_uuid: "f6027b6d2046425cad68647ce6c49497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29142a4fd630400d9c0d022689a38fd1"
I20260924 20:18:39.510452 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfce2e723184412b535e01feaf96b26" candidate_uuid: "f6027b6d2046425cad68647ce6c49497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55610796e8db4437b5b56d13bbb3030c"
I20260924 20:18:39.510471 14146 raft_consensus.cc:3037] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:39.510598 14414 raft_consensus.cc:3037] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:39.512044 14146 raft_consensus.cc:2454] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6027b6d2046425cad68647ce6c49497 in term 1.
I20260924 20:18:39.512315 14414 raft_consensus.cc:2454] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6027b6d2046425cad68647ce6c49497 in term 1.
I20260924 20:18:39.512418 14214 leader_election.cc:302] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29142a4fd630400d9c0d022689a38fd1, f6027b6d2046425cad68647ce6c49497; no voters:
I20260924 20:18:39.512604 14484 raft_consensus.cc:2781] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:39.512892 14484 raft_consensus.cc:686] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 1 LEADER]: Becoming Leader. State: Replica: f6027b6d2046425cad68647ce6c49497, State: Running, Role: LEADER
I20260924 20:18:39.513310 14484 consensus_queue.cc:237] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } }
I20260924 20:18:39.517370 14000 catalog_manager.cc:5495] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6027b6d2046425cad68647ce6c49497 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "f6027b6d2046425cad68647ce6c49497" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6027b6d2046425cad68647ce6c49497" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 39623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:39.595497 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.591550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.29.83.130:39623)
0924 20:18:39.591683 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.591686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.591700 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:39.591842 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.591844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.591901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.591998 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:39.592304 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.592814 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.593846 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.594076 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.594169 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.594560 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:39.595018 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:39.595118 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.595318 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.595381 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:18:39.602295 14414 raft_consensus.cc:1261] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 1 FOLLOWER]: Refusing update from remote peer f6027b6d2046425cad68647ce6c49497: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:39.602295 14146 raft_consensus.cc:1261] T cdfce2e723184412b535e01feaf96b26 P 29142a4fd630400d9c0d022689a38fd1 [term 1 FOLLOWER]: Refusing update from remote peer f6027b6d2046425cad68647ce6c49497: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:39.603169 14484 consensus_queue.cc:1035] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55610796e8db4437b5b56d13bbb3030c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:39.603411 14488 consensus_queue.cc:1035] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29142a4fd630400d9c0d022689a38fd1" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 38503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:39.618029 14491 mvcc.cc:205] Tried to move back new op lower bound from 7332991465882488832 to 7332991465530105856. Current Snapshot: MvccSnapshot[applied={T|T < 7332991465882488832 or (T in {7332991465882488832})}]
I20260924 20:18:39.621734 14001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:39.621950 14001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:39.622975 14492 mvcc.cc:205] Tried to move back new op lower bound from 7332991465882488832 to 7332991465530105856. Current Snapshot: MvccSnapshot[applied={T|T < 7332991465882488832}]
I20260924 20:18:39.624716 14001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68aabe64e2794c2a97b5e145edcc9fd3: Sending DeleteTablet for 3 replicas of tablet cdfce2e723184412b535e01feaf96b26
I20260924 20:18:39.625535 14394 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfce2e723184412b535e01feaf96b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:58554
I20260924 20:18:39.625792 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfce2e723184412b535e01feaf96b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:58838
I20260924 20:18:39.625849 14126 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfce2e723184412b535e01feaf96b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:59634
I20260924 20:18:39.626138 14125 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfce2e723184412b535e01feaf96b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:39 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:59634
I20260924 20:18:39.626179 14501 tablet_replica.cc:330] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: stopping tablet replica
I20260924 20:18:39.626582 14501 raft_consensus.cc:2229] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:39.626965 14501 raft_consensus.cc:2258] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:39.627265 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14063
I20260924 20:18:39.629429 14501 ts_tablet_manager.cc:1905] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:39.631807 14501 ts_tablet_manager.cc:1918] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:39.631919 14501 log.cc:1196] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/cdfce2e723184412b535e01feaf96b26
I20260924 20:18:39.632192 14501 ts_tablet_manager.cc:1939] T cdfce2e723184412b535e01feaf96b26 P 55610796e8db4437b5b56d13bbb3030c: Deleting consensus metadata
I20260924 20:18:39.633430 13988 catalog_manager.cc:4841] TS 55610796e8db4437b5b56d13bbb3030c (127.29.83.131:44449): tablet cdfce2e723184412b535e01feaf96b26 (table test-table [id=01d8ca3b129243b5ab2faf71526729ac]) successfully deleted
I20260924 20:18:39.633955 14502 tablet_replica.cc:330] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: stopping tablet replica
I20260924 20:18:39.634255 14502 raft_consensus.cc:2229] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:39.634588 14502 raft_consensus.cc:2258] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:39.635773 14502 ts_tablet_manager.cc:1905] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:18:39.638019 13986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:38503 (error 108)
I20260924 20:18:39.638159 14502 ts_tablet_manager.cc:1918] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:39.638249 14502 log.cc:1196] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/cdfce2e723184412b535e01feaf96b26
I20260924 20:18:39.638597 14502 ts_tablet_manager.cc:1939] T cdfce2e723184412b535e01feaf96b26 P f6027b6d2046425cad68647ce6c49497: Deleting consensus metadata
I20260924 20:18:39.639442 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14197
I20260924 20:18:39.639498 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.639193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:38503 (local address 127.0.0.1:59640)
0924 20:18:39.639339 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.639412 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:38503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 20:18:39.640317 13986 catalog_manager.cc:4584] TS 29142a4fd630400d9c0d022689a38fd1 (127.29.83.129:38503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdfce2e723184412b535e01feaf96b26: Network error: Client connection negotiation failed: client connection to 127.29.83.129:38503: connect: Connection refused (error 111)
I20260924 20:18:39.641218 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.640989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:38503 (local address 127.0.0.1:59650)
0924 20:18:39.641102 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.641158 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:38503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 20:18:39.646152 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14331
I20260924 20:18:39.647396 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.646387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:39623 (local address 127.0.0.1:58858)
0924 20:18:39.647234 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:39.647298 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.130:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748}
I20260924 20:18:39.651695 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 13969
2026-09-24T20:18:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2762 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T20:18:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:39Z Disabled control of system clock
I20260924 20:18:39.682976 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38657
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:38657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:39.799974 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:39.800263 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:39.800299 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:39.804432 14508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:39.804507 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:39.804522 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:39.804540 14508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:39.804556 14508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:39.809515 14508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:38657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:39.810724 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:39.812191 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:39.821753 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:39.821794 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:39.821803 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:39.822162 14508 server_base.cc:1034] running on GCE node
I20260924 20:18:39.822906 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:39.824115 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:39.825321 14508 hybrid_clock.cc:648] HybridClock initialized: now 1790281119825305 us; error 57 us; skew 500 ppm
I20260924 20:18:39.834728 14508 webserver.cc:460] Webserver started at http://127.29.83.190:46643/ using document root <none> and password file <none>
I20260924 20:18:39.835665 14508 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:39.835793 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:39.836053 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:39.838420 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "493f813dc583405c8ab2203cdb43bb82"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "8ff2af02c53514536744055ec10ef9f0"
server_key_iv: "82660be4a1dd28022fe060b65d809b86"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.839025 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "493f813dc583405c8ab2203cdb43bb82"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "8ff2af02c53514536744055ec10ef9f0"
server_key_iv: "82660be4a1dd28022fe060b65d809b86"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.843044 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:18:39.845865 14522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:39.847272 14508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:39.847440 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "493f813dc583405c8ab2203cdb43bb82"
format_stamp: "Formatted at 2026-09-24 20:18:39 on dist-test-slave-52fr"
server_key: "8ff2af02c53514536744055ec10ef9f0"
server_key_iv: "82660be4a1dd28022fe060b65d809b86"
server_key_version: "encryptionkey@0"
I20260924 20:18:39.847575 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:39.860638 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:39.861404 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:39.861618 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:39.868803 14508 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:38657
I20260924 20:18:39.868813 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:38657 every 8 connection(s)
I20260924 20:18:39.869887 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:39.870203 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14508
I20260924 20:18:39.870342 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:39.870838 30030 external_mini_cluster.cc:1428] Setting key a5d88528ef1f3e794d6e2f74eb24d3da
I20260924 20:18:39.873701 14575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:39.879351 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: Bootstrap starting.
I20260924 20:18:39.880527 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:39.872288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.29.83.190:38657)
0924 20:18:39.872987 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:39.873003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:39.873039 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:18:39.873693 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:39.873709 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:39.874190 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:39.874571 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:39.874998 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.875965 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.876864 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:39.877693 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:39.877961 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:39.878877 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:39.878882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:39.878896 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:39.879280 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:39.879333 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:39.879352 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:39.879457 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:39.879591 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 20:18:39.879848 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260924 20:18:39.881927 14575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:39.882716 14575 log.cc:824] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:39.884646 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: No bootstrap required, opened a new log
I20260924 20:18:39.888253 14575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } }
I20260924 20:18:39.888479 14575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:39.888514 14575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493f813dc583405c8ab2203cdb43bb82, State: Initialized, Role: FOLLOWER
I20260924 20:18:39.888923 14575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } }
I20260924 20:18:39.889043 14575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:39.889088 14575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:39.889158 14575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:39.890093 14575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } }
I20260924 20:18:39.890385 14575 leader_election.cc:302] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493f813dc583405c8ab2203cdb43bb82; no voters:
I20260924 20:18:39.890631 14575 leader_election.cc:288] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:39.890791 14580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:39.891052 14580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [term 1 LEADER]: Becoming Leader. State: Replica: 493f813dc583405c8ab2203cdb43bb82, State: Running, Role: LEADER
I20260924 20:18:39.891355 14580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } }
I20260924 20:18:39.891835 14575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:39.892958 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493f813dc583405c8ab2203cdb43bb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } } }
I20260924 20:18:39.893107 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:39.893460 14589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:39.894013 14582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493f813dc583405c8ab2203cdb43bb82. Latest consensus state: current_term: 1 leader_uuid: "493f813dc583405c8ab2203cdb43bb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493f813dc583405c8ab2203cdb43bb82" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 38657 } } }
I20260924 20:18:39.894117 14582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:39.896807 14589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:39.903055 14589 catalog_manager.cc:1347] Generated new cluster ID: f8b192a8ceef46ca86cb8513a0c988b5
I20260924 20:18:39.903153 14589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:39.923094 14589 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:39.924503 14589 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:39.935565 14589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: Generated new TSK 0
I20260924 20:18:39.936501 14589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:39.945209 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--builtin_ntp_servers=127.29.83.148:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:40.062680 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:40.062968 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:40.063009 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:40.067202 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:40.067267 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:40.067325 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:40.072521 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:40.073789 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:40.075269 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:40.085464 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:40.085455 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:40.085624 14599 server_base.cc:1034] running on GCE node
W20260924 20:18:40.085474 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:40.086226 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:40.086937 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:40.088121 14599 hybrid_clock.cc:648] HybridClock initialized: now 1790281120088088 us; error 46 us; skew 500 ppm
I20260924 20:18:40.090037 14599 webserver.cc:460] Webserver started at http://127.29.83.129:44305/ using document root <none> and password file <none>
I20260924 20:18:40.090605 14599 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:40.090696 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:40.090934 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:40.093302 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "4a3571f9008b1cbcb1d1ed50cc5fa0ec"
server_key_iv: "1f4ebe6122e6c1a4327476f35769ef1f"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.094018 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "4a3571f9008b1cbcb1d1ed50cc5fa0ec"
server_key_iv: "1f4ebe6122e6c1a4327476f35769ef1f"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.098137 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:18:40.100832 14613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.102315 14599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:40.102473 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "4a3571f9008b1cbcb1d1ed50cc5fa0ec"
server_key_iv: "1f4ebe6122e6c1a4327476f35769ef1f"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.102602 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:40.124897 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:40.125644 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:40.125917 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:40.126538 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:40.127518 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:40.127590 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.127660 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:40.127710 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.136866 14599 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:39969
I20260924 20:18:40.136874 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:39969 every 8 connection(s)
I20260924 20:18:40.137874 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:40.142694 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14599
I20260924 20:18:40.142846 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:18:40.143142 30030 external_mini_cluster.cc:1428] Setting key 601f5bd32aa136969bfbc77ae6758ac6
I20260924 20:18:40.145390 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--builtin_ntp_servers=127.29.83.148:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:40.148720 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.139761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:46405 (local address 127.29.83.190:38657)
0924 20:18:40.139928 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.139931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.141037 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 20:18:40.142500 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.142503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.142591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.142685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:40.143808 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.144471 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.145976 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.146219 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.146354 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.147840 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:40.147844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:40.147846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:40.147861 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:40.147883 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:40.147892 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:40.148048 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.148362 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.148488 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 20:18:40.149487 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.140159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38657 (local address 127.29.83.129:46405)
0924 20:18:40.140854 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.140895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.142255 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.142909 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.142926 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.143037 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:40.143642 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.143662 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.144642 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.144649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.145845 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.145856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.146393 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.146398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.146589 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.147403 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:40.147406 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:40.147472 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:40.147572 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:40.148055 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:40.148063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:40.148068 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.148217 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.148480 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260924 20:18:40.150908 14727 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38657
I20260924 20:18:40.151177 14727 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:40.151743 14727 heartbeater.cc:511] Master 127.29.83.190:38657 requested a full tablet report, sending...
I20260924 20:18:40.153370 14539 ts_manager.cc:194] Registered new tserver with Master: 3d7afb20d31f4dbfb04be9fd85e95d9d (127.29.83.129:39969)
I20260924 20:18:40.154892 14539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.129:46405
W20260924 20:18:40.260725 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:40.260988 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:40.261027 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:40.265110 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:40.265177 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:40.265233 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:40.270210 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:40.271374 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:40.272909 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:40.282636 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:40.282675 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:40.282881 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:40.282958 14730 server_base.cc:1034] running on GCE node
I20260924 20:18:40.283488 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:40.284106 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:40.285317 14730 hybrid_clock.cc:648] HybridClock initialized: now 1790281120285256 us; error 68 us; skew 500 ppm
I20260924 20:18:40.287384 14730 webserver.cc:460] Webserver started at http://127.29.83.130:34901/ using document root <none> and password file <none>
I20260924 20:18:40.287984 14730 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:40.288050 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:40.288259 14730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:40.290469 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "c0dc5fbadc3843d99884b435eaf5a035"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "00f3bbfd7028b1b89e738ab6124aa0b8"
server_key_iv: "cbc42e5ab8a0eb4ba3415e33d22bd9c6"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.291061 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "c0dc5fbadc3843d99884b435eaf5a035"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "00f3bbfd7028b1b89e738ab6124aa0b8"
server_key_iv: "cbc42e5ab8a0eb4ba3415e33d22bd9c6"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.295252 14730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:40.297978 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.299278 14730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:40.299422 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "c0dc5fbadc3843d99884b435eaf5a035"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "00f3bbfd7028b1b89e738ab6124aa0b8"
server_key_iv: "cbc42e5ab8a0eb4ba3415e33d22bd9c6"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.299546 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:40.326817 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:40.327595 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:40.327806 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:40.328440 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:40.329418 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:40.329466 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.329505 14730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:40.329555 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.340112 14730 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:35439
I20260924 20:18:40.340142 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:35439 every 8 connection(s)
I20260924 20:18:40.341077 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:40.343580 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14730
I20260924 20:18:40.343722 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:40.344008 30030 external_mini_cluster.cc:1428] Setting key 2ad991d75a029b92b459a09c38608a92
I20260924 20:18:40.346261 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--builtin_ntp_servers=127.29.83.148:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:40.354331 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.343046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:55315 (local address 127.29.83.190:38657)
0924 20:18:40.343425 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.343430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.344958 (+ 1528us) server_negotiation.cc:407] Connection header received
0924 20:18:40.346942 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.346948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.347023 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.347128 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:40.348438 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.349316 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.351281 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.351717 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.351814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.353589 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:40.353594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:40.353598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:40.353617 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:40.353649 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:40.353665 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:40.353777 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.354052 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.354142 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260924 20:18:40.356794 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.343531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38657 (local address 127.29.83.130:55315)
0924 20:18:40.344800 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.344843 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.346630 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.347389 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.347402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.347529 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:40.348282 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.348304 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.349471 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.349477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.351113 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.351129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.352060 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.352067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.352312 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.353212 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:40.353216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:40.353282 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:40.353385 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:40.353796 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:40.353809 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:40.353819 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.354031 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.354415 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":147,"threads_started":1}
I20260924 20:18:40.359467 14858 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38657
I20260924 20:18:40.359803 14858 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:40.360574 14858 heartbeater.cc:511] Master 127.29.83.190:38657 requested a full tablet report, sending...
I20260924 20:18:40.361938 14539 ts_manager.cc:194] Registered new tserver with Master: c0dc5fbadc3843d99884b435eaf5a035 (127.29.83.130:35439)
I20260924 20:18:40.362711 14539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.130:55315
W20260924 20:18:40.469496 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:40.469887 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:40.469971 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:40.474419 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:40.474507 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:40.474599 14861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:40.479599 14861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:40.480767 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:40.482364 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:40.493093 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:40.493139 14867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:40.493507 14861 server_base.cc:1034] running on GCE node
W20260924 20:18:40.493106 14869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:40.494159 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:40.494876 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:40.496109 14861 hybrid_clock.cc:648] HybridClock initialized: now 1790281120496068 us; error 54 us; skew 500 ppm
I20260924 20:18:40.498453 14861 webserver.cc:460] Webserver started at http://127.29.83.131:36069/ using document root <none> and password file <none>
I20260924 20:18:40.499117 14861 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:40.499224 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:40.499508 14861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:40.502182 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "ffa843503f064006ab6b772a9d4c5f49"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "2d53994e1f1f8ecfd93a3005742b5a11"
server_key_iv: "39d8f1541028ff54445b8a55dd449768"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.502894 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "ffa843503f064006ab6b772a9d4c5f49"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "2d53994e1f1f8ecfd93a3005742b5a11"
server_key_iv: "39d8f1541028ff54445b8a55dd449768"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.507427 14861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:40.510432 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.511790 14861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:40.511970 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "ffa843503f064006ab6b772a9d4c5f49"
format_stamp: "Formatted at 2026-09-24 20:18:40 on dist-test-slave-52fr"
server_key: "2d53994e1f1f8ecfd93a3005742b5a11"
server_key_iv: "39d8f1541028ff54445b8a55dd449768"
server_key_version: "encryptionkey@0"
I20260924 20:18:40.512102 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:40.530661 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:40.531425 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:40.531631 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:40.532245 14861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:40.533221 14861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:40.533293 14861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.533365 14861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:40.533413 14861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:40.542478 14861 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:40625
I20260924 20:18:40.542552 14988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:40625 every 8 connection(s)
I20260924 20:18:40.543464 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:40.544174 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 14861
I20260924 20:18:40.544279 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:18:40.544623 30030 external_mini_cluster.cc:1428] Setting key 0779b3643535a4e5f3101a2f5e01703b
I20260924 20:18:40.553995 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.545702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:35823 (local address 127.29.83.190:38657)
0924 20:18:40.545852 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.545857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.547192 (+ 1335us) server_negotiation.cc:407] Connection header received
0924 20:18:40.548632 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.548635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.548690 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.548786 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:40.549905 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.550486 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.551751 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.552060 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.552188 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.553371 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:40.553374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:40.553376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:40.553390 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:40.553409 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:40.553417 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:40.553483 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.553738 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.553870 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:18:40.554927 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.546255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:38657 (local address 127.29.83.131:35823)
0924 20:18:40.547041 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.547094 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.548398 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.548997 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.549004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.549096 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:40.549662 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.549677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.550643 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.550646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.551600 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.551610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.552246 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.552249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.552413 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.553049 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:40.553051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:40.553097 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:40.553179 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:40.553535 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:40.553544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:40.553550 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.553697 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.554030 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":141,"threads_started":1}
I20260924 20:18:40.556288 14989 heartbeater.cc:347] Connected to a master server at 127.29.83.190:38657
I20260924 20:18:40.556546 14989 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:40.557103 14989 heartbeater.cc:511] Master 127.29.83.190:38657 requested a full tablet report, sending...
I20260924 20:18:40.558367 14539 ts_manager.cc:194] Registered new tserver with Master: ffa843503f064006ab6b772a9d4c5f49 (127.29.83.131:40625)
I20260924 20:18:40.558966 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:40.559145 14539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.131:35823
I20260924 20:18:40.565310 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.561190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.29.83.190:38657)
0924 20:18:40.561343 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.561347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.561668 (+ 321us) server_negotiation.cc:407] Connection header received
0924 20:18:40.561889 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.561892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.561935 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.561997 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:40.562250 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.562759 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.563701 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.563955 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.564069 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.564886 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:40.564889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:40.564891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:40.564902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:40.564921 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:40.564929 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:40.564994 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.565083 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.565132 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:18:40.568637 14539 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:40.571859 14539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:40.587024 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.580258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35439 (local address 127.0.0.1:46768)
0924 20:18:40.581084 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.581141 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.581287 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.582054 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.582060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.582082 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:40.582231 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.582237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.583706 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.583712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.585321 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.585334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.586481 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.586484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.586601 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.586644 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.586728 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.586852 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":160,"thread_start_us":81,"threads_started":1}
I20260924 20:18:40.587736 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.580448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.29.83.130:35439)
0924 20:18:40.581473 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.581486 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.581512 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:18:40.581591 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.581600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.581848 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.582058 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:40.582385 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.583608 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.585474 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.586350 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.587237 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.587406 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.587455 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.587568 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":128,"threads_started":1}
I20260924 20:18:40.587793 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.580671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39969 (local address 127.0.0.1:44410)
0924 20:18:40.581677 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.581690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.581815 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.582504 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.582508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.582529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:40.582606 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.582613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.583612 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.583619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.586069 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.586080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.587404 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.587407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.587492 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.587513 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.587578 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.587661 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":886,"spinlock_wait_cycles":6656,"thread_start_us":104,"threads_started":1}
I20260924 20:18:40.588244 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.580469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:40625 (local address 127.0.0.1:41894)
0924 20:18:40.581663 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.581676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.581820 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.582327 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.582330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.582348 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:40.582449 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.582459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.584284 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.584287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.585930 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.585945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.587710 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.587717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.587851 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.587884 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.587990 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.588083 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"spinlock_wait_cycles":4224,"thread_start_us":103,"threads_started":1}
I20260924 20:18:40.588274 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.581087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.29.83.129:39969)
0924 20:18:40.581574 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.581586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.581815 (+ 229us) server_negotiation.cc:407] Connection header received
0924 20:18:40.582007 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.582015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.582215 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.582360 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:40.582728 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.583494 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.586210 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.587001 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.587088 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.587299 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.588028 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.588129 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":98,"threads_started":1}
I20260924 20:18:40.588840 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.580999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.29.83.131:40625)
0924 20:18:40.581573 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.581583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.581885 (+ 302us) server_negotiation.cc:407] Connection header received
0924 20:18:40.581971 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.581977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.582172 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.582346 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:40.582561 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.584130 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.586095 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.587038 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.587426 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.587639 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.588499 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.588623 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":121,"threads_started":1}
I20260924 20:18:40.591009 14792 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 (DEFAULT_TABLE table=test-table [id=4b9df71c4ab041c7b40c93a16da55130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:40.591009 14661 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 (DEFAULT_TABLE table=test-table [id=4b9df71c4ab041c7b40c93a16da55130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:40.592211 14661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b776be30d407ca10f12e9ff2e0b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:40.592211 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b776be30d407ca10f12e9ff2e0b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:40.595427 14922 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 (DEFAULT_TABLE table=test-table [id=4b9df71c4ab041c7b40c93a16da55130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:40.596644 14922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b776be30d407ca10f12e9ff2e0b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:40.599825 15006 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Bootstrap starting.
I20260924 20:18:40.603572 15007 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Bootstrap starting.
I20260924 20:18:40.603722 15006 tablet_bootstrap.cc:654] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:40.604874 15006 log.cc:824] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:40.606211 15009 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Bootstrap starting.
I20260924 20:18:40.606540 15007 tablet_bootstrap.cc:654] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:40.607420 15006 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: No bootstrap required, opened a new log
I20260924 20:18:40.607529 15007 log.cc:824] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:40.607677 15006 ts_tablet_manager.cc:1397] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 20:18:40.608916 15009 tablet_bootstrap.cc:654] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:40.609969 15007 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: No bootstrap required, opened a new log
I20260924 20:18:40.610112 15009 log.cc:824] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:40.610198 15007 ts_tablet_manager.cc:1397] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:18:40.611943 15006 raft_consensus.cc:348] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.612221 15006 raft_consensus.cc:374] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:40.612282 15006 raft_consensus.cc:729] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7afb20d31f4dbfb04be9fd85e95d9d, State: Initialized, Role: FOLLOWER
I20260924 20:18:40.612353 15009 tablet_bootstrap.cc:492] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: No bootstrap required, opened a new log
I20260924 20:18:40.612571 15009 ts_tablet_manager.cc:1397] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:18:40.612759 15006 consensus_queue.cc:260] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.613502 14727 heartbeater.cc:503] Master 127.29.83.190:38657 was elected leader, sending a full tablet report...
I20260924 20:18:40.613926 15006 ts_tablet_manager.cc:1428] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260924 20:18:40.614607 15007 raft_consensus.cc:348] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.614890 15007 raft_consensus.cc:374] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:40.614959 15007 raft_consensus.cc:729] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dc5fbadc3843d99884b435eaf5a035, State: Initialized, Role: FOLLOWER
I20260924 20:18:40.615424 15007 consensus_queue.cc:260] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.616544 15007 ts_tablet_manager.cc:1428] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:18:40.616904 14858 heartbeater.cc:503] Master 127.29.83.190:38657 was elected leader, sending a full tablet report...
I20260924 20:18:40.617738 15009 raft_consensus.cc:348] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.618265 15009 raft_consensus.cc:374] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:40.618403 15009 raft_consensus.cc:729] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa843503f064006ab6b772a9d4c5f49, State: Initialized, Role: FOLLOWER
I20260924 20:18:40.618978 15009 consensus_queue.cc:260] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.619743 14989 heartbeater.cc:503] Master 127.29.83.190:38657 was elected leader, sending a full tablet report...
I20260924 20:18:40.620098 15009 ts_tablet_manager.cc:1428] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
W20260924 20:18:40.639056 14728 tablet.cc:2369] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:40.794723 14990 tablet.cc:2369] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:40.829694 15012 raft_consensus.cc:482] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:40.830014 15012 raft_consensus.cc:504] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.831233 15012 leader_election.cc:288] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0dc5fbadc3843d99884b435eaf5a035 (127.29.83.130:35439), ffa843503f064006ab6b772a9d4c5f49 (127.29.83.131:40625)
I20260924 20:18:40.834976 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.831569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:40279 (local address 127.29.83.130:35439)
0924 20:18:40.831757 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.831762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.831861 (+ 99us) server_negotiation.cc:407] Connection header received
0924 20:18:40.832028 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.832031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.832083 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.832169 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:40.832449 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.832956 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834029 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.834415 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834542 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.834594 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.834786 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.834860 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:18:40.834995 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.831434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:35439 (local address 127.29.83.129:40279)
0924 20:18:40.831750 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.831766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.831905 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.832182 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.832184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.832200 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:40.832336 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.832342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.833073 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.833077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.833904 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.833912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.834543 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.834659 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.834678 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.834769 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.834841 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":75,"threads_started":1}
I20260924 20:18:40.835428 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.831639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:43155 (local address 127.29.83.131:40625)
0924 20:18:40.831750 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.831755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.832058 (+ 303us) server_negotiation.cc:407] Connection header received
0924 20:18:40.832198 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.832202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.832257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.832350 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:40.832650 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.833139 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834321 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.834750 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834907 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.834963 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.835213 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.835293 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:18:40.835628 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b776be30d407ca10f12e9ff2e0b20" candidate_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dc5fbadc3843d99884b435eaf5a035" is_pre_election: true
I20260924 20:18:40.835918 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.831566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:40625 (local address 127.29.83.129:43155)
0924 20:18:40.831945 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.831957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:40.832096 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:40.832361 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:40.832364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:40.832381 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:40.832527 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.832533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.833306 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.833311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.834187 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:40.834197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.834907 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.834911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:40.835013 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.835064 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:18:40.835615 (+ 551us) client_negotiation.cc:240] Negotiation successful
0924 20:18:40.835721 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260924 20:18:40.835976 14812 raft_consensus.cc:2454] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7afb20d31f4dbfb04be9fd85e95d9d in term 0.
I20260924 20:18:40.836531 14615 leader_election.cc:302] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7afb20d31f4dbfb04be9fd85e95d9d, c0dc5fbadc3843d99884b435eaf5a035; no voters:
I20260924 20:18:40.836635 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b776be30d407ca10f12e9ff2e0b20" candidate_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa843503f064006ab6b772a9d4c5f49" is_pre_election: true
I20260924 20:18:40.836826 15012 raft_consensus.cc:2781] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:40.836930 15012 raft_consensus.cc:482] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:40.836995 15012 raft_consensus.cc:3037] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:40.836968 14943 raft_consensus.cc:2454] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7afb20d31f4dbfb04be9fd85e95d9d in term 0.
I20260924 20:18:40.838372 15012 raft_consensus.cc:504] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
I20260924 20:18:40.838734 15012 leader_election.cc:288] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [CANDIDATE]: Term 1 election: Requested vote from peers c0dc5fbadc3843d99884b435eaf5a035 (127.29.83.130:35439), ffa843503f064006ab6b772a9d4c5f49 (127.29.83.131:40625)
I20260924 20:18:40.839140 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b776be30d407ca10f12e9ff2e0b20" candidate_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa843503f064006ab6b772a9d4c5f49"
I20260924 20:18:40.839138 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b776be30d407ca10f12e9ff2e0b20" candidate_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dc5fbadc3843d99884b435eaf5a035"
I20260924 20:18:40.839294 14943 raft_consensus.cc:3037] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:40.839262 14812 raft_consensus.cc:3037] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:40.840410 14943 raft_consensus.cc:2454] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7afb20d31f4dbfb04be9fd85e95d9d in term 1.
I20260924 20:18:40.840595 14812 raft_consensus.cc:2454] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7afb20d31f4dbfb04be9fd85e95d9d in term 1.
I20260924 20:18:40.840763 14617 leader_election.cc:302] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7afb20d31f4dbfb04be9fd85e95d9d, ffa843503f064006ab6b772a9d4c5f49; no voters:
I20260924 20:18:40.840981 15012 raft_consensus.cc:2781] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:40.841253 15012 raft_consensus.cc:686] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 1 LEADER]: Becoming Leader. State: Replica: 3d7afb20d31f4dbfb04be9fd85e95d9d, State: Running, Role: LEADER
I20260924 20:18:40.841590 15012 consensus_queue.cc:237] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } }
W20260924 20:18:40.842239 14859 tablet.cc:2369] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:40.844990 14537 catalog_manager.cc:5495] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d7afb20d31f4dbfb04be9fd85e95d9d (127.29.83.129). New cstate: current_term: 1 leader_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7afb20d31f4dbfb04be9fd85e95d9d" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 39969 } health_report { overall_health: HEALTHY } } }
I20260924 20:18:40.888978 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.885262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.29.83.129:39969)
0924 20:18:40.885425 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:40.885428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:40.885496 (+ 68us) server_negotiation.cc:407] Connection header received
0924 20:18:40.885627 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:40.885629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:40.885689 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:40.885875 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:40.886112 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.886630 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.887476 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:40.887690 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:40.887768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:40.888119 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:40.888580 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:40.888681 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:40.888766 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 20:18:40.888814 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 20:18:40.895524 14812 raft_consensus.cc:1261] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 1 FOLLOWER]: Refusing update from remote peer 3d7afb20d31f4dbfb04be9fd85e95d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:40.895529 14943 raft_consensus.cc:1261] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 1 FOLLOWER]: Refusing update from remote peer 3d7afb20d31f4dbfb04be9fd85e95d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:40.896292 15012 consensus_queue.cc:1035] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa843503f064006ab6b772a9d4c5f49" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:40.896507 15017 consensus_queue.cc:1035] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dc5fbadc3843d99884b435eaf5a035" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:40.907258 15022 mvcc.cc:205] Tried to move back new op lower bound from 7332991471180222464 to 7332991470970503168. Current Snapshot: MvccSnapshot[applied={T|T < 7332991471180222464}]
I20260924 20:18:40.913111 14539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:40.913300 14539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:40.915864 14539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493f813dc583405c8ab2203cdb43bb82: Sending DeleteTablet for 3 replicas of tablet ad3b776be30d407ca10f12e9ff2e0b20
I20260924 20:18:40.916716 14922 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:40 UTC) from {username='slave'} at 127.0.0.1:41894
I20260924 20:18:40.916716 14661 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:40 UTC) from {username='slave'} at 127.0.0.1:44410
I20260924 20:18:40.916710 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b776be30d407ca10f12e9ff2e0b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:40 UTC) from {username='slave'} at 127.0.0.1:46768
I20260924 20:18:40.917265 15031 tablet_replica.cc:330] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: stopping tablet replica
I20260924 20:18:40.917387 15029 tablet_replica.cc:330] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d: stopping tablet replica
I20260924 20:18:40.917616 15031 raft_consensus.cc:2229] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:40.917614 15029 raft_consensus.cc:2229] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:40.917985 15031 raft_consensus.cc:2258] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:40.917985 15029 raft_consensus.cc:2258] T ad3b776be30d407ca10f12e9ff2e0b20 P 3d7afb20d31f4dbfb04be9fd85e95d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:40.918404 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14599
I20260924 20:18:40.919557 15030 tablet_replica.cc:330] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: stopping tablet replica
I20260924 20:18:40.919898 15030 raft_consensus.cc:2229] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:40.920199 15030 raft_consensus.cc:2258] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:40.921304 15031 ts_tablet_manager.cc:1905] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:40.921691 15030 ts_tablet_manager.cc:1905] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:40.923522 15031 ts_tablet_manager.cc:1918] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:40.923640 15031 log.cc:1196] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/ad3b776be30d407ca10f12e9ff2e0b20
I20260924 20:18:40.923938 15031 ts_tablet_manager.cc:1939] T ad3b776be30d407ca10f12e9ff2e0b20 P c0dc5fbadc3843d99884b435eaf5a035: Deleting consensus metadata
I20260924 20:18:40.925171 15030 ts_tablet_manager.cc:1918] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:40.925192 14524 catalog_manager.cc:4841] TS c0dc5fbadc3843d99884b435eaf5a035 (127.29.83.130:35439): tablet ad3b776be30d407ca10f12e9ff2e0b20 (table test-table [id=4b9df71c4ab041c7b40c93a16da55130]) successfully deleted
I20260924 20:18:40.925295 15030 log.cc:1196] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/ad3b776be30d407ca10f12e9ff2e0b20
I20260924 20:18:40.925625 15030 ts_tablet_manager.cc:1939] T ad3b776be30d407ca10f12e9ff2e0b20 P ffa843503f064006ab6b772a9d4c5f49: Deleting consensus metadata
W20260924 20:18:40.926357 14525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:39969 (error 108)
I20260924 20:18:40.926962 14526 catalog_manager.cc:4841] TS ffa843503f064006ab6b772a9d4c5f49 (127.29.83.131:40625): tablet ad3b776be30d407ca10f12e9ff2e0b20 (table test-table [id=4b9df71c4ab041c7b40c93a16da55130]) successfully deleted
I20260924 20:18:40.927388 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14730
I20260924 20:18:40.927558 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:40.927175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:39969 (local address 127.0.0.1:44416)
0924 20:18:40.927359 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:40.927430 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 20:18:40.928886 14525 catalog_manager.cc:4584] TS 3d7afb20d31f4dbfb04be9fd85e95d9d (127.29.83.129:39969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad3b776be30d407ca10f12e9ff2e0b20: Network error: Client connection negotiation failed: client connection to 127.29.83.129:39969: connect: Connection refused (error 111)
I20260924 20:18:40.933458 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14861
I20260924 20:18:40.939091 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 14508
2026-09-24T20:18:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1286 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:41 dist-test-slave-52fr krb5kdc[15035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:41 dist-test-slave-52fr krb5kdc[15035](info): set up 2 sockets
Sep 24 20:18:41 dist-test-slave-52fr krb5kdc[15035](info): commencing operation
krb5kdc: starting...
W20260924 20:18:41.985689 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:18:42.113267 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35745
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:35745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:42.222988 15048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:42.223272 15048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:42.223320 15048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:42.227385 15048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:42.227452 15048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:42.227473 15048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:42.227491 15048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:42.227506 15048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:42.232903 15048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:35745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:42.234050 15048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:42.235092 15048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:42.244023 15053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:42.244017 15056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:42.244023 15054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.244102 15048 server_base.cc:1034] running on GCE node
I20260924 20:18:42.244865 15048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:42.246109 15048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:42.247308 15048 hybrid_clock.cc:648] HybridClock initialized: now 1790281122247277 us; error 47 us; skew 500 ppm
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:42.250404 15048 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:18:42.251317 15048 webserver.cc:460] Webserver started at http://127.29.83.190:43659/ using document root <none> and password file <none>
I20260924 20:18:42.251852 15048 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:42.251916 15048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:42.252086 15048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:42.254237 15048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "6d4f4fbde72545a4a58a595e0d7201c1"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "5de6e9de1454b98ee617cd37c3d6d707"
server_key_iv: "0b27cf049366a1128c3dc0b3ef831830"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.254793 15048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "6d4f4fbde72545a4a58a595e0d7201c1"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "5de6e9de1454b98ee617cd37c3d6d707"
server_key_iv: "0b27cf049366a1128c3dc0b3ef831830"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.258800 15048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:18:42.261556 15063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.262971 15048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:18:42.263134 15048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "6d4f4fbde72545a4a58a595e0d7201c1"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "5de6e9de1454b98ee617cd37c3d6d707"
server_key_iv: "0b27cf049366a1128c3dc0b3ef831830"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.263255 15048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:42.278745 15048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:42.282188 15048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:42.282397 15048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:42.289424 15048 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:35745
I20260924 20:18:42.289533 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:35745 every 8 connection(s)
I20260924 20:18:42.290449 15048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:42.293229 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:42.298810 15116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: Bootstrap starting.
I20260924 20:18:42.299621 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15048
I20260924 20:18:42.299762 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:42.300065 30030 external_mini_cluster.cc:1428] Setting key 77ccc3f43e7e93a4cc3de71de9fcfd2d
I20260924 20:18:42.301822 15116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:42.302700 15116 log.cc:824] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:42.304913 15116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: No bootstrap required, opened a new log
I20260924 20:18:42.309394 15116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } }
I20260924 20:18:42.309681 15116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:42.309741 15116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4f4fbde72545a4a58a595e0d7201c1, State: Initialized, Role: FOLLOWER
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:42.310248 15116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } }
I20260924 20:18:42.310436 15116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:42.310501 15116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:42.310598 15116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:42.311766 15116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } }
I20260924 20:18:42.312160 15116 leader_election.cc:302] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d4f4fbde72545a4a58a595e0d7201c1; no voters:
I20260924 20:18:42.312528 15116 leader_election.cc:288] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:42.312644 15121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:42.312901 15121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4f4fbde72545a4a58a595e0d7201c1, State: Running, Role: LEADER
I20260924 20:18:42.313293 15121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } }
I20260924 20:18:42.313862 15116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:42.315083 15123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } } }
I20260924 20:18:42.315214 15123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:42.315639 15122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4f4fbde72545a4a58a595e0d7201c1. Latest consensus state: current_term: 1 leader_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4f4fbde72545a4a58a595e0d7201c1" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35745 } } }
I20260924 20:18:42.315815 15122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:42.317075 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:42.301381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.29.83.190:35745)
0924 20:18:42.301861 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:42.301882 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:42.301923 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:18:42.302808 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:42.302832 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:42.303249 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:42.303603 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:42.304289 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.305746 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.306773 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.307513 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.307690 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:42.310889 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:42.310896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:42.310912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:42.310942 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:42.313108 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.313640 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.313645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.313652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.313817 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.314119 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.314122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.314124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.314659 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:42.314877 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:42.315801 (+ 924us) server_negotiation.cc:299] Negotiation successful
0924 20:18:42.316126 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
W20260924 20:18:42.319352 15136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:18:42.319486 15136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:18:42.319569 15130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:42.320277 15130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:42.325459 15130 catalog_manager.cc:1347] Generated new cluster ID: 0cb5bf6002d64592a9c767a79689dcd2
I20260924 20:18:42.325521 15130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:42.346617 15130 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:42.347632 15130 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:42.354812 15130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: Generated new TSK 0
I20260924 20:18:42.355660 15130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:18:42.394012 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:42.508633 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:42.508963 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:42.509042 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:42.513197 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:42.513329 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:42.513486 15142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:42.519165 15142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:42.520246 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:42.521278 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:42.531373 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:42.531366 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.531515 15142 server_base.cc:1034] running on GCE node
W20260924 20:18:42.531373 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.532087 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:42.532737 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:42.533914 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790281122533891 us; error 27 us; skew 500 ppm
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:42.536732 15142 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:18:42.537688 15142 webserver.cc:460] Webserver started at http://127.29.83.129:46543/ using document root <none> and password file <none>
I20260924 20:18:42.538318 15142 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:42.538383 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:42.538592 15142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:42.540638 15142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "f761db85fd934d459e048f018c826415"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2d0442f09a2aeb61bc761944e7ab92b9"
server_key_iv: "874a7d60d78502b1c81a1fe7652579c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.541203 15142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "f761db85fd934d459e048f018c826415"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2d0442f09a2aeb61bc761944e7ab92b9"
server_key_iv: "874a7d60d78502b1c81a1fe7652579c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.545207 15142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:18:42.547773 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.548974 15142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:42.549090 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "f761db85fd934d459e048f018c826415"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2d0442f09a2aeb61bc761944e7ab92b9"
server_key_iv: "874a7d60d78502b1c81a1fe7652579c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.549201 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:42.565685 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:42.569311 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:42.569518 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:42.570189 15142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:42.571029 15142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:42.571075 15142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.571144 15142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:42.571174 15142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.580291 15142 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:37173
I20260924 20:18:42.580309 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:37173 every 8 connection(s)
I20260924 20:18:42.581189 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:42.591536 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15142
I20260924 20:18:42.591675 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:18:42.591966 30030 external_mini_cluster.cc:1428] Setting key 072e68dab000c14b965c336ecd81b893
I20260924 20:18:42.596491 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:42.583132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:48133 (local address 127.29.83.190:35745)
0924 20:18:42.583286 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:42.583296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:42.584067 (+ 771us) server_negotiation.cc:407] Connection header received
0924 20:18:42.585422 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:42.585425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:42.585482 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:42.585589 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:42.586829 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.587447 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.588706 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.588962 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.589082 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:42.591903 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:42.591907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:42.591909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:42.591936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:42.593671 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.594456 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.594459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.594460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.594538 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.595025 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.595028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.595029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.595237 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:42.595320 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:42.596074 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 20:18:42.596276 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:18:42.597177 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:42.583455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35745 (local address 127.29.83.129:48133)
0924 20:18:42.583915 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:42.583958 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:42.585201 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:42.585814 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:42.585823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:42.586196 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:42.586671 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:42.586685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.587622 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.587626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:42.588570 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:42.588580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.589122 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.589126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:42.589306 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:42.590118 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:42.590135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:42.591688 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:42.593846 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:42.593853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:42.593869 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:42.594318 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:42.594637 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:42.594644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:42.594649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:42.594872 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:42.595351 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:42.595361 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:42.595734 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 20:18:42.595945 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 20:18:42.596241 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260924 20:18:42.598610 15271 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35745
I20260924 20:18:42.598882 15271 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:42.599540 15271 heartbeater.cc:511] Master 127.29.83.190:35745 requested a full tablet report, sending...
I20260924 20:18:42.601243 15080 ts_manager.cc:194] Registered new tserver with Master: f761db85fd934d459e048f018c826415 (127.29.83.129:37173)
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:18:42.602547 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:48133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:18:42.621209 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:42.732650 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:42.732918 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:42.732960 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:42.737051 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:42.737116 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:42.737201 15276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:42.742636 15276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:42.743742 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:42.744877 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:42.755113 15281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.755332 15276 server_base.cc:1034] running on GCE node
W20260924 20:18:42.755110 15282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:42.755164 15284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.755999 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:42.756647 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:42.757848 15276 hybrid_clock.cc:648] HybridClock initialized: now 1790281122757831 us; error 27 us; skew 500 ppm
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:42.760641 15276 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:18:42.761615 15276 webserver.cc:460] Webserver started at http://127.29.83.130:32937/ using document root <none> and password file <none>
I20260924 20:18:42.762197 15276 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:42.762259 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:42.762416 15276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:42.764398 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "30106ed88b8f4f26b234960dc8f23721"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "67f00ce0a9332c32c2c804a7c69eb4b7"
server_key_iv: "b50cd0efd9628a4778804b6f15daf5c0"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.764918 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "30106ed88b8f4f26b234960dc8f23721"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "67f00ce0a9332c32c2c804a7c69eb4b7"
server_key_iv: "b50cd0efd9628a4778804b6f15daf5c0"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.768930 15276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:18:42.771453 15291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.772706 15276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:18:42.772805 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "30106ed88b8f4f26b234960dc8f23721"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "67f00ce0a9332c32c2c804a7c69eb4b7"
server_key_iv: "b50cd0efd9628a4778804b6f15daf5c0"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.772892 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:42.789016 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:42.792793 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:42.793000 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:42.793649 15276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:42.794538 15276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:42.794587 15276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.794626 15276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:42.794683 15276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.803789 15276 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:37729
I20260924 20:18:42.803851 15404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:37729 every 8 connection(s)
I20260924 20:18:42.804783 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:42.807915 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15276
I20260924 20:18:42.808099 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:42.808385 30030 external_mini_cluster.cc:1428] Setting key 4dda26ca83190618e8e22e8decb49e9d
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:18:42.819590 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:42.806766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:51019 (local address 127.29.83.190:35745)
0924 20:18:42.806909 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:42.806919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:42.807687 (+ 768us) server_negotiation.cc:407] Connection header received
0924 20:18:42.809177 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:42.809181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:42.809235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:42.809379 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:42.810664 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.811190 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.812358 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.812583 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.812695 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:42.815531 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:42.815535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:42.815537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:42.815571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:42.817307 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.817869 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.817872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.817874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.817955 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:42.818311 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:42.818314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:42.818316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:42.818510 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:42.818678 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:42.819204 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 20:18:42.819342 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
I20260924 20:18:42.820484 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:42.807054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35745 (local address 127.29.83.130:51019)
0924 20:18:42.807521 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:42.807567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:42.808961 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:42.809571 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:42.809579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:42.810024 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:42.810518 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:42.810532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.811338 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.811347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:42.812229 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:42.812238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:42.812739 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:42.812742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:42.812920 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:42.813581 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:42.813598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:42.815338 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:42.817434 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:42.817440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:42.817458 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:42.817726 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:42.818049 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:42.818052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:42.818054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:42.818206 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:42.818677 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:42.818685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:42.818961 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 20:18:42.819154 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 20:18:42.819525 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260924 20:18:42.821801 15405 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35745
I20260924 20:18:42.822098 15405 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:42.822636 15405 heartbeater.cc:511] Master 127.29.83.190:35745 requested a full tablet report, sending...
I20260924 20:18:42.823855 15080 ts_manager.cc:194] Registered new tserver with Master: 30106ed88b8f4f26b234960dc8f23721 (127.29.83.130:37729)
I20260924 20:18:42.824739 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:51019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:18:42.838733 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:42.950613 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:42.950994 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:42.951057 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:42.955261 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:42.955334 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:42.955425 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:42.960956 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:42.962173 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:42.963158 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:42.973601 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.973675 15410 server_base.cc:1034] running on GCE node
W20260924 20:18:42.973544 15416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:42.973551 15415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:42.974349 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:42.975104 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:42.976280 15410 hybrid_clock.cc:648] HybridClock initialized: now 1790281122976239 us; error 51 us; skew 500 ppm
Sep 24 20:18:42 dist-test-slave-52fr krb5kdc[15035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:42.979719 15410 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:18:42.980666 15410 webserver.cc:460] Webserver started at http://127.29.83.131:45861/ using document root <none> and password file <none>
I20260924 20:18:42.981266 15410 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:42.981370 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:42.981619 15410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:42.983783 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "6d081aa58e1b44ee80e2845f29520be3"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2949356fb50578b18c7e2579a87c4219"
server_key_iv: "23c01d99530dd85a331856fe2a33a20e"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.984380 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "6d081aa58e1b44ee80e2845f29520be3"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2949356fb50578b18c7e2579a87c4219"
server_key_iv: "23c01d99530dd85a331856fe2a33a20e"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.988278 15410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:18:42.991153 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:42.992815 15410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:42.992980 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "6d081aa58e1b44ee80e2845f29520be3"
format_stamp: "Formatted at 2026-09-24 20:18:42 on dist-test-slave-52fr"
server_key: "2949356fb50578b18c7e2579a87c4219"
server_key_iv: "23c01d99530dd85a331856fe2a33a20e"
server_key_version: "encryptionkey@0"
I20260924 20:18:42.993131 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:43.005178 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:43.008723 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:43.008957 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:43.009584 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:43.010499 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:43.010572 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.010646 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:43.010699 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.020188 15410 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:33103
I20260924 20:18:43.020218 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:33103 every 8 connection(s)
I20260924 20:18:43.021082 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:43.025699 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15410
I20260924 20:18:43.025880 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:18:43.026199 30030 external_mini_cluster.cc:1428] Setting key 03631f459f2f529ba6540f5382566833
Sep 24 20:18:43 dist-test-slave-52fr krb5kdc[15035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281122, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:43.035292 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.023041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:56241 (local address 127.29.83.190:35745)
0924 20:18:43.023226 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.023231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.024086 (+ 855us) server_negotiation.cc:407] Connection header received
0924 20:18:43.025334 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.025337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.025386 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.025466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:43.026983 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.027532 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.028705 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.028937 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.029049 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.031597 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:43.031607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:43.031610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:43.031635 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:43.033131 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:43.033659 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:43.033663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:43.033664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:43.033747 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:43.034146 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:43.034150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:43.034152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:43.034337 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:43.034444 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.034959 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.035115 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 20:18:43.036037 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.023478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35745 (local address 127.29.83.131:56241)
0924 20:18:43.023931 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.023973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.025116 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.025702 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.025712 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.026253 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:43.026838 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.026852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.027665 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.027668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.028577 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.028586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.029084 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.029088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.029253 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.029892 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:43.029907 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:43.031431 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:43.033265 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:43.033271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:43.033287 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:43.033547 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:43.033901 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:43.033904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:43.033906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:43.034049 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:43.034441 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:43.034448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:43.034689 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.034883 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.035220 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260924 20:18:43.037144 15539 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35745
I20260924 20:18:43.037364 15539 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:43.037868 15539 heartbeater.cc:511] Master 127.29.83.190:35745 requested a full tablet report, sending...
I20260924 20:18:43.038887 15080 ts_manager.cc:194] Registered new tserver with Master: 6d081aa58e1b44ee80e2845f29520be3 (127.29.83.131:33103)
I20260924 20:18:43.039501 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:56241
I20260924 20:18:43.040169 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:43.050906 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.042476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.29.83.190:35745)
0924 20:18:43.042616 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.042620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.042864 (+ 244us) server_negotiation.cc:407] Connection header received
0924 20:18:43.042936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.042938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.042978 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.043052 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:43.043821 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.044341 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.045226 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.045474 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.045569 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.047255 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:43.047259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:43.047261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:43.047287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:43.048833 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:43.049306 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:43.049309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:43.049310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:43.049412 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:43.049742 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:43.049744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:43.049746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:43.050122 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:43.050241 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.050531 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.050701 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:43.054472 15077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:43.057596 15077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:43.072513 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.066112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37173 (local address 127.0.0.1:53814)
0924 20:18:43.066534 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.066581 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.066720 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.068007 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.068014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.068041 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.068196 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.068202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.069429 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.069436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.070917 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.070927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.071947 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.071951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.072061 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072112 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.072219 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.072345 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":19,"thread_start_us":88,"threads_started":1}
I20260924 20:18:43.072822 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.066867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:33103 (local address 127.0.0.1:50426)
0924 20:18:43.067222 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.067237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.067336 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.068377 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.068380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.068395 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.068472 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.068478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.069918 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.069921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.071227 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.071237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.072177 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.072181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.072274 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072300 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.072398 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.072471 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":46,"thread_start_us":127,"threads_started":1}
I20260924 20:18:43.073014 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.067255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:37729 (local address 127.0.0.1:58008)
0924 20:18:43.067979 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.067992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.068085 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.068584 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.068587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.068607 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.068691 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.068696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.070190 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.070193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.071331 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.071341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.072620 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.072626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.072732 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072754 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.072821 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.072901 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
I20260924 20:18:43.072906 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.066206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.29.83.129:37173)
0924 20:18:43.066472 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.066497 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.066653 (+ 156us) server_negotiation.cc:407] Connection header received
0924 20:18:43.067529 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.067538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.067790 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.068356 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.068505 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.069292 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.071047 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.071792 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.071915 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072071 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.072439 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.072554 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260924 20:18:43.073434 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.067630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.29.83.130:37729)
0924 20:18:43.067995 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.068008 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.068056 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:18:43.068228 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.068234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.068406 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.068841 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.068981 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.070051 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.071451 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.072321 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.072424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072556 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.073155 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.073276 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":137,"threads_started":1}
I20260924 20:18:43.073699 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.067103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.29.83.131:33103)
0924 20:18:43.067888 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.067900 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.067924 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:18:43.068024 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.068032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.068232 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.068391 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.068575 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.069723 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.071381 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.072065 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.072175 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.072333 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.072579 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.073061 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":150,"threads_started":1}
I20260924 20:18:43.076079 15339 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f (DEFAULT_TABLE table=test-table [id=cc1c7790ea994171b84e2fc122b7a595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:43.076161 15205 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f (DEFAULT_TABLE table=test-table [id=cc1c7790ea994171b84e2fc122b7a595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:43.076159 15473 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f (DEFAULT_TABLE table=test-table [id=cc1c7790ea994171b84e2fc122b7a595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:43.077170 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:43.077170 15205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:43.077178 15339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:43.083990 15556 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Bootstrap starting.
I20260924 20:18:43.085700 15557 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Bootstrap starting.
I20260924 20:18:43.086102 15558 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: Bootstrap starting.
I20260924 20:18:43.086985 15556 tablet_bootstrap.cc:654] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:43.087859 15558 tablet_bootstrap.cc:654] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:43.087916 15557 tablet_bootstrap.cc:654] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:43.088598 15556 log.cc:824] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:43.089138 15558 log.cc:824] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:43.089139 15557 log.cc:824] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:43.090986 15558 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: No bootstrap required, opened a new log
I20260924 20:18:43.090987 15556 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: No bootstrap required, opened a new log
I20260924 20:18:43.091203 15558 ts_tablet_manager.cc:1397] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:18:43.091205 15556 ts_tablet_manager.cc:1397] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:18:43.092326 15557 tablet_bootstrap.cc:492] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: No bootstrap required, opened a new log
I20260924 20:18:43.092515 15557 ts_tablet_manager.cc:1397] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:18:43.094318 15558 raft_consensus.cc:348] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.094579 15558 raft_consensus.cc:374] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:43.094631 15558 raft_consensus.cc:729] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f761db85fd934d459e048f018c826415, State: Initialized, Role: FOLLOWER
I20260924 20:18:43.095069 15558 consensus_queue.cc:260] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.095206 15556 raft_consensus.cc:348] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.095482 15556 raft_consensus.cc:374] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:43.095538 15556 raft_consensus.cc:729] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30106ed88b8f4f26b234960dc8f23721, State: Initialized, Role: FOLLOWER
I20260924 20:18:43.096019 15556 consensus_queue.cc:260] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.096256 15558 ts_tablet_manager.cc:1428] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:18:43.096428 15557 raft_consensus.cc:348] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.096616 15271 heartbeater.cc:503] Master 127.29.83.190:35745 was elected leader, sending a full tablet report...
I20260924 20:18:43.096652 15557 raft_consensus.cc:374] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:43.096704 15557 raft_consensus.cc:729] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d081aa58e1b44ee80e2845f29520be3, State: Initialized, Role: FOLLOWER
I20260924 20:18:43.096988 15556 ts_tablet_manager.cc:1428] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:18:43.097119 15557 consensus_queue.cc:260] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.097334 15405 heartbeater.cc:503] Master 127.29.83.190:35745 was elected leader, sending a full tablet report...
I20260924 20:18:43.098076 15557 ts_tablet_manager.cc:1428] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:18:43.098526 15539 heartbeater.cc:503] Master 127.29.83.190:35745 was elected leader, sending a full tablet report...
I20260924 20:18:43.121184 15564 raft_consensus.cc:482] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:43.121411 15564 raft_consensus.cc:504] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.122607 15564 leader_election.cc:288] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f761db85fd934d459e048f018c826415 (127.29.83.129:37173), 30106ed88b8f4f26b234960dc8f23721 (127.29.83.130:37729)
I20260924 20:18:43.127699 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.122936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:54605 (local address 127.29.83.129:37173)
0924 20:18:43.123077 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.123082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.123097 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:18:43.123237 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.123240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.123294 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.123374 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.123785 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.124642 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.126383 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.127009 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.127141 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.127193 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.127490 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.127577 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:18:43.127737 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.122803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37173 (local address 127.29.83.131:54605)
0924 20:18:43.122950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.122965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.123085 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.123491 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.123495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.123517 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.123653 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.123660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.124772 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.124775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.126193 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.126204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.127141 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.127145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.127327 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.127391 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.127461 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.127561 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":116}
I20260924 20:18:43.127912 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.123232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:46613 (local address 127.29.83.130:37729)
0924 20:18:43.123397 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.123406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.123584 (+ 178us) server_negotiation.cc:407] Connection header received
0924 20:18:43.123672 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.123674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.123741 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.123842 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.124102 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.124808 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.126384 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.127007 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.127141 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.127193 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.127490 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.127796 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 20:18:43.127749 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.123118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:37729 (local address 127.29.83.131:46613)
0924 20:18:43.123495 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.123508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.123592 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.123868 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.123870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.123886 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.123992 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.123996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.124932 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.124936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.126193 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.126204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.127141 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.127145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.127325 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.127365 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.127451 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.127561 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":85,"thread_start_us":149,"threads_started":1}
I20260924 20:18:43.128196 15563 raft_consensus.cc:482] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:43.128391 15563 raft_consensus.cc:504] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.128711 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "6d081aa58e1b44ee80e2845f29520be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30106ed88b8f4f26b234960dc8f23721" is_pre_election: true
I20260924 20:18:43.128875 15225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "6d081aa58e1b44ee80e2845f29520be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f761db85fd934d459e048f018c826415" is_pre_election: true
I20260924 20:18:43.128926 15359 raft_consensus.cc:2454] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d081aa58e1b44ee80e2845f29520be3 in term 0.
I20260924 20:18:43.129143 15225 raft_consensus.cc:2454] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d081aa58e1b44ee80e2845f29520be3 in term 0.
I20260924 20:18:43.129477 15428 leader_election.cc:302] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30106ed88b8f4f26b234960dc8f23721, 6d081aa58e1b44ee80e2845f29520be3; no voters:
I20260924 20:18:43.129662 15563 leader_election.cc:288] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f761db85fd934d459e048f018c826415 (127.29.83.129:37173), 6d081aa58e1b44ee80e2845f29520be3 (127.29.83.131:33103)
I20260924 20:18:43.129841 15564 raft_consensus.cc:2781] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:43.129941 15564 raft_consensus.cc:482] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:43.129987 15564 raft_consensus.cc:3037] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:43.131681 15564 raft_consensus.cc:504] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.132076 15564 leader_election.cc:288] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [CANDIDATE]: Term 1 election: Requested vote from peers f761db85fd934d459e048f018c826415 (127.29.83.129:37173), 30106ed88b8f4f26b234960dc8f23721 (127.29.83.130:37729)
I20260924 20:18:43.132429 15225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "6d081aa58e1b44ee80e2845f29520be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f761db85fd934d459e048f018c826415"
I20260924 20:18:43.132447 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "6d081aa58e1b44ee80e2845f29520be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30106ed88b8f4f26b234960dc8f23721"
I20260924 20:18:43.132570 15359 raft_consensus.cc:3037] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:43.132571 15225 raft_consensus.cc:3037] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:43.133901 15359 raft_consensus.cc:2454] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d081aa58e1b44ee80e2845f29520be3 in term 1.
I20260924 20:18:43.133901 15225 raft_consensus.cc:2454] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d081aa58e1b44ee80e2845f29520be3 in term 1.
I20260924 20:18:43.134244 15428 leader_election.cc:302] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d081aa58e1b44ee80e2845f29520be3, f761db85fd934d459e048f018c826415; no voters:
I20260924 20:18:43.134555 15564 raft_consensus.cc:2781] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:43.134775 15564 raft_consensus.cc:686] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 1 LEADER]: Becoming Leader. State: Replica: 6d081aa58e1b44ee80e2845f29520be3, State: Running, Role: LEADER
I20260924 20:18:43.134814 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.130315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:58419 (local address 127.29.83.131:33103)
0924 20:18:43.130436 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.130442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.130456 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:43.130576 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.130580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.130628 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.130707 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.131108 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.131811 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.133498 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.134135 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.134221 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.134268 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.134612 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.134689 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:18:43.134858 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.130100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:33103 (local address 127.29.83.130:58419)
0924 20:18:43.130387 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.130400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.130485 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.130732 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.130735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.130773 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.130986 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.130991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.131909 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.131912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.133318 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.133334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.134321 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.134324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.134428 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.134454 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.134575 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.134659 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":55,"spinlock_wait_cycles":25728,"thread_start_us":139,"threads_started":1}
I20260924 20:18:43.135169 15564 consensus_queue.cc:237] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } }
I20260924 20:18:43.135460 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "30106ed88b8f4f26b234960dc8f23721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d081aa58e1b44ee80e2845f29520be3" is_pre_election: true
I20260924 20:18:43.135756 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.129846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37173 (local address 127.29.83.130:46671)
0924 20:18:43.130002 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.130019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.130105 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.130732 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.130735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.130775 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:43.130985 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.130991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.131740 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.131747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.134357 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.134390 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.135339 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.135353 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.135460 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.135490 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.135566 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.135641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":40,"spinlock_wait_cycles":6144}
I20260924 20:18:43.136173 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.130428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:46671 (local address 127.29.83.129:37173)
0924 20:18:43.130540 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.130546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.130557 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:18:43.130594 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.130596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.130634 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.130707 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:43.131121 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.131627 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.134509 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.135110 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.135224 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.135280 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.135989 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.136071 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 20:18:43.136536 15225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0fcc9ae4b4e4bb7e0615c72531a2f" candidate_uuid: "30106ed88b8f4f26b234960dc8f23721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f761db85fd934d459e048f018c826415" is_pre_election: true
I20260924 20:18:43.136708 15225 raft_consensus.cc:2379] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30106ed88b8f4f26b234960dc8f23721 in current term 1: Already voted for candidate 6d081aa58e1b44ee80e2845f29520be3 in this term.
I20260924 20:18:43.137183 15294 leader_election.cc:302] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30106ed88b8f4f26b234960dc8f23721; no voters: 6d081aa58e1b44ee80e2845f29520be3, f761db85fd934d459e048f018c826415
I20260924 20:18:43.137666 15563 raft_consensus.cc:2726] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 20:18:43.138749 15080 catalog_manager.cc:5495] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d081aa58e1b44ee80e2845f29520be3 (127.29.83.131). New cstate: current_term: 1 leader_uuid: "6d081aa58e1b44ee80e2845f29520be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d081aa58e1b44ee80e2845f29520be3" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 33103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:43.154453 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.150352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.29.83.131:33103)
0924 20:18:43.150485 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.150489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.150503 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:43.150598 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.150600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.150652 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.150739 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:43.151023 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.151680 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.152692 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.152921 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.153012 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.153367 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:43.153914 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:43.154031 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.154177 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.154240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:43.160238 15225 raft_consensus.cc:1261] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P f761db85fd934d459e048f018c826415 [term 1 FOLLOWER]: Refusing update from remote peer 6d081aa58e1b44ee80e2845f29520be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:43.160295 15359 raft_consensus.cc:1261] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 1 FOLLOWER]: Refusing update from remote peer 6d081aa58e1b44ee80e2845f29520be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:43.160986 15564 consensus_queue.cc:1035] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f761db85fd934d459e048f018c826415" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:43.161175 15567 consensus_queue.cc:1035] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30106ed88b8f4f26b234960dc8f23721" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:43.171310 15571 mvcc.cc:205] Tried to move back new op lower bound from 7332991480457322496 to 7332991480364974080. Current Snapshot: MvccSnapshot[applied={T|T < 7332991480457322496}]
I20260924 20:18:43.177002 15080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:43.177182 15080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:43.179816 15080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d4f4fbde72545a4a58a595e0d7201c1: Sending DeleteTablet for 3 replicas of tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f
I20260924 20:18:43.180725 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:43 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:50426
I20260924 20:18:43.180727 15205 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:43 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:53814
I20260924 20:18:43.181226 15579 tablet_replica.cc:330] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: stopping tablet replica
I20260924 20:18:43.181227 15339 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:43 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:58008
I20260924 20:18:43.181667 15579 raft_consensus.cc:2229] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:43.182049 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15142
I20260924 20:18:43.182999 15581 tablet_replica.cc:330] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: stopping tablet replica
I20260924 20:18:43.183323 15581 raft_consensus.cc:2229] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:43.183656 15581 raft_consensus.cc:2258] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:43.183848 15579 raft_consensus.cc:2258] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:43.185070 15581 ts_tablet_manager.cc:1905] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:43.187345 15581 ts_tablet_manager.cc:1918] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:43.187461 15581 log.cc:1196] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/c9f0fcc9ae4b4e4bb7e0615c72531a2f
I20260924 20:18:43.188167 15581 ts_tablet_manager.cc:1939] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 30106ed88b8f4f26b234960dc8f23721: Deleting consensus metadata
I20260924 20:18:43.189155 15579 ts_tablet_manager.cc:1905] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:43.189399 15066 catalog_manager.cc:4841] TS 30106ed88b8f4f26b234960dc8f23721 (127.29.83.130:37729): tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f (table test-table [id=cc1c7790ea994171b84e2fc122b7a595]) successfully deleted
W20260924 20:18:43.189881 15066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:37173 (error 108)
I20260924 20:18:43.190006 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15276
I20260924 20:18:43.191809 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.191528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:37173 (local address 127.0.0.1:53830)
0924 20:18:43.191650 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.191719 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:18:43.191992 15066 catalog_manager.cc:4584] TS f761db85fd934d459e048f018c826415 (127.29.83.129:37173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f: Network error: Client connection negotiation failed: client connection to 127.29.83.129:37173: connect: Connection refused (error 111)
I20260924 20:18:43.193542 15579 ts_tablet_manager.cc:1918] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:43.193658 15579 log.cc:1196] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/c9f0fcc9ae4b4e4bb7e0615c72531a2f
I20260924 20:18:43.193980 15579 ts_tablet_manager.cc:1939] T c9f0fcc9ae4b4e4bb7e0615c72531a2f P 6d081aa58e1b44ee80e2845f29520be3: Deleting consensus metadata
I20260924 20:18:43.195015 15066 catalog_manager.cc:4841] TS 6d081aa58e1b44ee80e2845f29520be3 (127.29.83.131:33103): tablet c9f0fcc9ae4b4e4bb7e0615c72531a2f (table test-table [id=cc1c7790ea994171b84e2fc122b7a595]) successfully deleted
I20260924 20:18:43.197299 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15410
I20260924 20:18:43.204895 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15048
2026-09-24T20:18:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2266 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T20:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:43Z Disabled control of system clock
I20260924 20:18:43.235496 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41457
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:43.349154 15586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:43.349447 15586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:43.349483 15586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:43.353623 15586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:43.353693 15586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:43.353709 15586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:43.353727 15586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:43.353744 15586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:43.358690 15586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:43.359731 15586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:43.361135 15586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:43.370365 15591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:43.370463 15586 server_base.cc:1034] running on GCE node
W20260924 20:18:43.370286 15594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:43.370286 15592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:43.371243 15586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:43.372361 15586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:43.373564 15586 hybrid_clock.cc:648] HybridClock initialized: now 1790281123373535 us; error 43 us; skew 500 ppm
I20260924 20:18:43.375643 15586 webserver.cc:460] Webserver started at http://127.29.83.190:42813/ using document root <none> and password file <none>
I20260924 20:18:43.376260 15586 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:43.376327 15586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:43.376574 15586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:43.378724 15586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "89b2c534e49a4386b5b6748c1c24279a"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0b8e6bf1f3af10074562a8a0435374d2"
server_key_iv: "a160a7ed20bd3df54e7cb7c3e481c5ae"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.379356 15586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "89b2c534e49a4386b5b6748c1c24279a"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0b8e6bf1f3af10074562a8a0435374d2"
server_key_iv: "a160a7ed20bd3df54e7cb7c3e481c5ae"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.383180 15586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:43.385730 15600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.387094 15586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:43.387245 15586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "89b2c534e49a4386b5b6748c1c24279a"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0b8e6bf1f3af10074562a8a0435374d2"
server_key_iv: "a160a7ed20bd3df54e7cb7c3e481c5ae"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.387365 15586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:43.397220 15586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:43.397940 15586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:43.398123 15586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:43.404898 15586 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:41457
I20260924 20:18:43.404935 15652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:41457 every 8 connection(s)
I20260924 20:18:43.405933 15586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:43.409242 15653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:43.411470 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15586
I20260924 20:18:43.411612 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:43.411916 30030 external_mini_cluster.cc:1428] Setting key 21a441dbd9853a2d6f48828a69795ef8
I20260924 20:18:43.415930 15653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: Bootstrap starting.
I20260924 20:18:43.418275 15653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:43.419040 15653 log.cc:824] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:43.421012 15653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: No bootstrap required, opened a new log
I20260924 20:18:43.421639 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.413513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.29.83.190:41457)
0924 20:18:43.414240 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.414266 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.414303 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:18:43.415108 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.415129 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.415552 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.415924 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:43.416370 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.417322 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.418326 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.418860 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.419032 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.419902 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:43.419907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:43.419920 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:43.420314 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:43.420366 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:43.420386 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:43.420493 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.420614 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.420875 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":152,"threads_started":1}
I20260924 20:18:43.424947 15653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } }
I20260924 20:18:43.425204 15653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:43.425272 15653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b2c534e49a4386b5b6748c1c24279a, State: Initialized, Role: FOLLOWER
I20260924 20:18:43.425735 15653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } }
I20260924 20:18:43.425920 15653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:43.425971 15653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:43.426043 15653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:43.426908 15653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } }
I20260924 20:18:43.427196 15653 leader_election.cc:302] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b2c534e49a4386b5b6748c1c24279a; no voters:
I20260924 20:18:43.427472 15653 leader_election.cc:288] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:43.427575 15658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:43.427877 15658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [term 1 LEADER]: Becoming Leader. State: Replica: 89b2c534e49a4386b5b6748c1c24279a, State: Running, Role: LEADER
I20260924 20:18:43.428184 15658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } }
I20260924 20:18:43.428705 15653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:43.429803 15659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b2c534e49a4386b5b6748c1c24279a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } } }
I20260924 20:18:43.430110 15660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b2c534e49a4386b5b6748c1c24279a. Latest consensus state: current_term: 1 leader_uuid: "89b2c534e49a4386b5b6748c1c24279a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b2c534e49a4386b5b6748c1c24279a" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41457 } } }
I20260924 20:18:43.430226 15659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:43.430290 15660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:43.430740 15670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:43.433543 15670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:43.438921 15670 catalog_manager.cc:1347] Generated new cluster ID: f75306d3c74a46188cb024bca58a45b7
I20260924 20:18:43.438998 15670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:43.465054 15670 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:43.466230 15670 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:43.474247 15670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: Generated new TSK 0
I20260924 20:18:43.475207 15670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:43.488942 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--builtin_ntp_servers=127.29.83.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:43.605724 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:43.606071 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:43.606148 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:43.610313 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:43.610394 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:43.610486 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:43.615509 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:43.616731 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:43.618214 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:43.628084 15685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:43.628089 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:43.628481 15677 server_base.cc:1034] running on GCE node
W20260924 20:18:43.628504 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:43.629035 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:43.629863 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:43.631103 15677 hybrid_clock.cc:648] HybridClock initialized: now 1790281123631070 us; error 59 us; skew 500 ppm
I20260924 20:18:43.633041 15677 webserver.cc:460] Webserver started at http://127.29.83.129:35249/ using document root <none> and password file <none>
I20260924 20:18:43.633639 15677 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:43.633706 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:43.633955 15677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:43.636049 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "6555e830745049f2898319a196cd6eff"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "53ad316c0b98d04e7713642fd96e1da5"
server_key_iv: "b82779818d58b7005565b11c27690181"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.636606 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "6555e830745049f2898319a196cd6eff"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "53ad316c0b98d04e7713642fd96e1da5"
server_key_iv: "b82779818d58b7005565b11c27690181"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.640421 15677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:43.643153 15691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.644562 15677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:43.644721 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "6555e830745049f2898319a196cd6eff"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "53ad316c0b98d04e7713642fd96e1da5"
server_key_iv: "b82779818d58b7005565b11c27690181"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.644847 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:43.659186 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:43.659876 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:43.660070 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:43.660663 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:43.661597 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:43.661667 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.661738 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:43.661795 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.670825 15677 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:40809
I20260924 20:18:43.670843 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:40809 every 8 connection(s)
I20260924 20:18:43.671761 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:43.676348 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15677
I20260924 20:18:43.676461 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:18:43.676749 30030 external_mini_cluster.cc:1428] Setting key 79871b4621b2fa645d394e05f344378f
I20260924 20:18:43.678957 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--builtin_ntp_servers=127.29.83.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:43.683043 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.673551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:45613 (local address 127.29.83.190:41457)
0924 20:18:43.673698 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.673703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.675007 (+ 1304us) server_negotiation.cc:407] Connection header received
0924 20:18:43.676647 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.676651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.676720 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.676856 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:43.678117 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.678673 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.680725 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.681141 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.681265 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.682509 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:43.682513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:43.682515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:43.682529 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:43.682553 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:43.682563 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:43.682642 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.682841 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.682912 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:18:43.684341 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.674070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41457 (local address 127.29.83.129:45613)
0924 20:18:43.674807 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.674879 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.676322 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.677049 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.677057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.677171 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:43.677922 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.677945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.678819 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.678827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.680572 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.680588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.681336 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.681340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.681494 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.682138 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:43.682143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:43.682202 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:43.682290 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:43.682662 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:43.682674 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:43.682683 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.682807 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.683130 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260924 20:18:43.685725 15805 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41457
I20260924 20:18:43.686062 15805 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:43.686817 15805 heartbeater.cc:511] Master 127.29.83.190:41457 requested a full tablet report, sending...
I20260924 20:18:43.688634 15617 ts_manager.cc:194] Registered new tserver with Master: 6555e830745049f2898319a196cd6eff (127.29.83.129:40809)
I20260924 20:18:43.690100 15617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.129:45613
W20260924 20:18:43.794286 15808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:43.794597 15808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:43.794677 15808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:43.798816 15808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:43.798900 15808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:43.798992 15808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:43.804245 15808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:43.805591 15808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:43.807099 15808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:43.817184 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:43.817243 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:43.817181 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:43.817538 15808 server_base.cc:1034] running on GCE node
I20260924 20:18:43.818019 15808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:43.818647 15808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:43.819818 15808 hybrid_clock.cc:648] HybridClock initialized: now 1790281123819781 us; error 48 us; skew 500 ppm
I20260924 20:18:43.822000 15808 webserver.cc:460] Webserver started at http://127.29.83.130:46377/ using document root <none> and password file <none>
I20260924 20:18:43.822620 15808 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:43.822685 15808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:43.822906 15808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:43.825027 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "7ad6217c930b43268566e358654d6efe"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0a4b930173863ede3d7ee281a06f9f24"
server_key_iv: "fafbbeef3ee3cbdd1d8ac34546a4ff8c"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.825600 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "7ad6217c930b43268566e358654d6efe"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0a4b930173863ede3d7ee281a06f9f24"
server_key_iv: "fafbbeef3ee3cbdd1d8ac34546a4ff8c"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.829629 15808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:18:43.832342 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.833622 15808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:43.833802 15808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "7ad6217c930b43268566e358654d6efe"
format_stamp: "Formatted at 2026-09-24 20:18:43 on dist-test-slave-52fr"
server_key: "0a4b930173863ede3d7ee281a06f9f24"
server_key_iv: "fafbbeef3ee3cbdd1d8ac34546a4ff8c"
server_key_version: "encryptionkey@0"
I20260924 20:18:43.833940 15808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:43.852923 15808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:43.853699 15808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:43.853958 15808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:43.854585 15808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:43.855583 15808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:43.855657 15808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.855733 15808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:43.855777 15808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:43.865617 15808 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:41179
I20260924 20:18:43.865648 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:41179 every 8 connection(s)
I20260924 20:18:43.866654 15808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:43.876978 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15808
I20260924 20:18:43.877164 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:43.877476 30030 external_mini_cluster.cc:1428] Setting key 2061b92b59ac14f41754c8ab8a45b50e
I20260924 20:18:43.877697 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.869295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:57623 (local address 127.29.83.190:41457)
0924 20:18:43.869514 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:43.869517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:43.870768 (+ 1251us) server_negotiation.cc:407] Connection header received
0924 20:18:43.872278 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:43.872281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:43.872343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:43.872455 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:43.873461 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.874073 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.875315 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.875543 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.875662 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.877027 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:43.877031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:43.877033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:43.877047 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:43.877068 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:43.877077 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:43.877161 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:43.877365 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 20:18:43.877440 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 20:18:43.878993 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:43.869795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41457 (local address 127.29.83.130:57623)
0924 20:18:43.870580 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:43.870628 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:43.872029 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:43.872652 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:43.872659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:43.872753 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:43.873284 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.873300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.874234 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.874236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.875175 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:43.875184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:43.875709 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:43.875713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:43.875894 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:43.876695 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:43.876697 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:43.876746 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:43.876816 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:43.877162 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:43.877174 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:43.877183 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:18:43.877365 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 20:18:43.877865 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":140,"threads_started":1}
I20260924 20:18:43.879725 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--builtin_ntp_servers=127.29.83.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:43.880388 15936 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41457
I20260924 20:18:43.880656 15936 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:43.881337 15936 heartbeater.cc:511] Master 127.29.83.190:41457 requested a full tablet report, sending...
I20260924 20:18:43.882578 15617 ts_manager.cc:194] Registered new tserver with Master: 7ad6217c930b43268566e358654d6efe (127.29.83.130:41179)
I20260924 20:18:43.883175 15617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.130:57623
W20260924 20:18:43.993639 15939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:43.993986 15939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:43.994059 15939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:43.998159 15939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:43.998216 15939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:43.998296 15939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:44.003146 15939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:44.004258 15939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:44.005714 15939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:44.016124 15947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:44.016077 15945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:44.016105 15944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:44.016517 15939 server_base.cc:1034] running on GCE node
I20260924 20:18:44.016949 15939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:44.017696 15939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:44.018886 15939 hybrid_clock.cc:648] HybridClock initialized: now 1790281124018864 us; error 36 us; skew 500 ppm
I20260924 20:18:44.020925 15939 webserver.cc:460] Webserver started at http://127.29.83.131:39619/ using document root <none> and password file <none>
I20260924 20:18:44.021540 15939 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:44.021616 15939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:44.021893 15939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:44.024041 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "42f6382b8589408bbf49e1dcfd4d416c"
format_stamp: "Formatted at 2026-09-24 20:18:44 on dist-test-slave-52fr"
server_key: "6c52e17d0b327d1f412ffdc877736095"
server_key_iv: "23dd717b43028873523f409bd8e1c7c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:44.024628 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "42f6382b8589408bbf49e1dcfd4d416c"
format_stamp: "Formatted at 2026-09-24 20:18:44 on dist-test-slave-52fr"
server_key: "6c52e17d0b327d1f412ffdc877736095"
server_key_iv: "23dd717b43028873523f409bd8e1c7c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:44.028673 15939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:44.031450 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:44.032823 15939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:44.032969 15939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "42f6382b8589408bbf49e1dcfd4d416c"
format_stamp: "Formatted at 2026-09-24 20:18:44 on dist-test-slave-52fr"
server_key: "6c52e17d0b327d1f412ffdc877736095"
server_key_iv: "23dd717b43028873523f409bd8e1c7c7"
server_key_version: "encryptionkey@0"
I20260924 20:18:44.033092 15939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:44.050230 15939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:44.051007 15939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:44.051222 15939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:44.051860 15939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:44.052891 15939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:44.052963 15939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:44.053037 15939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:44.053087 15939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:44.062147 15939 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:35203
I20260924 20:18:44.062165 16066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:35203 every 8 connection(s)
I20260924 20:18:44.063112 15939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:44.067057 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 15939
I20260924 20:18:44.067188 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:18:44.067550 30030 external_mini_cluster.cc:1428] Setting key 4678cb57211857356b05d7e25d594abf
I20260924 20:18:44.073396 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.064973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:40629 (local address 127.29.83.190:41457)
0924 20:18:44.065113 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.065116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.066348 (+ 1232us) server_negotiation.cc:407] Connection header received
0924 20:18:44.068113 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.068117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.068181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.068259 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:44.069451 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.070179 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.071346 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.071557 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.071634 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.072743 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:44.072746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:44.072748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:44.072760 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:44.072779 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:44.072786 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:44.072865 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.073229 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.073283 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:44.074409 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.065427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:41457 (local address 127.29.83.131:40629)
0924 20:18:44.066165 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.066211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.067816 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.068469 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.068498 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.068614 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:44.069306 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.069320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.070307 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.070310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.071225 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.071236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.071729 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.071733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.071883 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.072488 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:44.072489 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:44.072532 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:44.072601 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:44.073079 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:44.073088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:44.073093 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.073214 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.073513 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260924 20:18:44.075614 16067 heartbeater.cc:347] Connected to a master server at 127.29.83.190:41457
I20260924 20:18:44.075892 16067 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:44.076452 16067 heartbeater.cc:511] Master 127.29.83.190:41457 requested a full tablet report, sending...
I20260924 20:18:44.077519 15617 ts_manager.cc:194] Registered new tserver with Master: 42f6382b8589408bbf49e1dcfd4d416c (127.29.83.131:35203)
I20260924 20:18:44.078177 15617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.131:40629
I20260924 20:18:44.081991 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:44.087999 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.084020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.29.83.190:41457)
0924 20:18:44.084192 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.084196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.084292 (+ 96us) server_negotiation.cc:407] Connection header received
0924 20:18:44.084445 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.084450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.084498 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.084566 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:44.084872 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.085397 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.086289 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.086510 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.086590 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.087387 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:44.087390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:44.087392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:44.087404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:44.087424 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:44.087432 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:44.087547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.087727 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.087814 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 20:18:44.091174 15617 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:44.094684 15617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:44.111089 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.105707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.29.83.130:41179)
0924 20:18:44.106140 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.106151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.106489 (+ 338us) server_negotiation.cc:407] Connection header received
0924 20:18:44.106567 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.106572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.106752 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.106932 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:44.107111 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.107837 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.109466 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.110275 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110387 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.110574 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.110770 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.110898 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":140,"threads_started":1}
I20260924 20:18:44.111084 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.104490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35203 (local address 127.0.0.1:54222)
0924 20:18:44.105445 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.105504 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.105637 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.106385 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.106395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.106423 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:44.106604 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.106610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.108137 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.108140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.109086 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.109096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.110432 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.110552 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.110601 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.110731 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.110923 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":137,"thread_start_us":131,"threads_started":1}
I20260924 20:18:44.111086 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:41179 (local address 127.0.0.1:60446)
0924 20:18:44.106366 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.106378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.106466 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.106907 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.106910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.106927 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:44.107011 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.107017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.107971 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.107977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.109335 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.109345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.110397 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.110540 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.110586 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.110738 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.110917 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":15,"thread_start_us":88,"threads_started":1}
I20260924 20:18:44.111598 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.104669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.29.83.131:35203)
0924 20:18:44.105599 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.105611 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.105636 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:18:44.105808 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.105813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.106032 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.106188 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:44.106722 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.107982 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.109231 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.110039 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110151 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.110357 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.111304 (+ 947us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.111415 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":133,"threads_started":1}
I20260924 20:18:44.112010 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.104750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40809 (local address 127.0.0.1:38704)
0924 20:18:44.105544 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.105560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.105747 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.106510 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.106514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.106532 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:44.106635 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.106641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.108106 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.108109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.109966 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.109976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.111628 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.111632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.111732 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.111755 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.111827 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.111893 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":66,"thread_start_us":130,"threads_started":1}
I20260924 20:18:44.114166 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.104988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.29.83.129:40809)
0924 20:18:44.105387 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.105400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.105657 (+ 257us) server_negotiation.cc:407] Connection header received
0924 20:18:44.105946 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.105951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.106154 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.106312 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:44.106797 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.107982 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110090 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.110854 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.110955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.111138 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.113887 (+ 2749us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.114005 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":96,"threads_started":1}
I20260924 20:18:44.114490 16001 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd8078f3dda469a86d082acff4faff9 (DEFAULT_TABLE table=test-table [id=a313fb6386cb4cf39722b1b12b53c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:44.114616 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd8078f3dda469a86d082acff4faff9 (DEFAULT_TABLE table=test-table [id=a313fb6386cb4cf39722b1b12b53c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:44.115520 16001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd8078f3dda469a86d082acff4faff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:44.115535 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd8078f3dda469a86d082acff4faff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:44.121722 16085 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Bootstrap starting.
I20260924 20:18:44.123193 16084 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Bootstrap starting.
I20260924 20:18:44.124046 16085 tablet_bootstrap.cc:654] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:44.124797 16085 log.cc:824] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:44.124820 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd8078f3dda469a86d082acff4faff9 (DEFAULT_TABLE table=test-table [id=a313fb6386cb4cf39722b1b12b53c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:44.125329 16084 tablet_bootstrap.cc:654] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:44.126080 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd8078f3dda469a86d082acff4faff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:44.126204 16084 log.cc:824] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:44.126920 16085 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: No bootstrap required, opened a new log
I20260924 20:18:44.127131 16085 ts_tablet_manager.cc:1397] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 20:18:44.128326 16084 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: No bootstrap required, opened a new log
I20260924 20:18:44.128516 16084 ts_tablet_manager.cc:1397] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:18:44.131857 16085 raft_consensus.cc:348] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.132159 16085 raft_consensus.cc:374] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:44.132241 16085 raft_consensus.cc:729] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f6382b8589408bbf49e1dcfd4d416c, State: Initialized, Role: FOLLOWER
I20260924 20:18:44.132721 16085 consensus_queue.cc:260] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.132918 16084 raft_consensus.cc:348] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.133185 16084 raft_consensus.cc:374] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:44.133244 16084 raft_consensus.cc:729] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad6217c930b43268566e358654d6efe, State: Initialized, Role: FOLLOWER
I20260924 20:18:44.133579 16067 heartbeater.cc:503] Master 127.29.83.190:41457 was elected leader, sending a full tablet report...
I20260924 20:18:44.133855 16085 ts_tablet_manager.cc:1428] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 20:18:44.134263 16088 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Bootstrap starting.
I20260924 20:18:44.134470 16084 consensus_queue.cc:260] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.135488 16084 ts_tablet_manager.cc:1428] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:18:44.135823 15936 heartbeater.cc:503] Master 127.29.83.190:41457 was elected leader, sending a full tablet report...
I20260924 20:18:44.136668 16088 tablet_bootstrap.cc:654] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:44.137684 16088 log.cc:824] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:44.139853 16088 tablet_bootstrap.cc:492] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: No bootstrap required, opened a new log
I20260924 20:18:44.140066 16088 ts_tablet_manager.cc:1397] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:18:44.143342 16088 raft_consensus.cc:348] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.143682 16088 raft_consensus.cc:374] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:44.143770 16088 raft_consensus.cc:729] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6555e830745049f2898319a196cd6eff, State: Initialized, Role: FOLLOWER
I20260924 20:18:44.144309 16088 consensus_queue.cc:260] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.145072 15805 heartbeater.cc:503] Master 127.29.83.190:41457 was elected leader, sending a full tablet report...
I20260924 20:18:44.145525 16088 ts_tablet_manager.cc:1428] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 20:18:44.173009 15806 tablet.cc:2369] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:44.314145 16068 tablet.cc:2369] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:18:44.367961 15937 tablet.cc:2369] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:44.377976 16090 raft_consensus.cc:482] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:44.378286 16090 raft_consensus.cc:504] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.379500 16090 leader_election.cc:288] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42f6382b8589408bbf49e1dcfd4d416c (127.29.83.131:35203), 6555e830745049f2898319a196cd6eff (127.29.83.129:40809)
I20260924 20:18:44.383744 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.379915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:33409 (local address 127.29.83.131:35203)
0924 20:18:44.380045 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.380049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.380214 (+ 165us) server_negotiation.cc:407] Connection header received
0924 20:18:44.380395 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.380400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.380444 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.380529 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:44.380809 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.381335 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.382596 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.383146 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.383287 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.383338 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.383536 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.383605 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:18:44.383770 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.379793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:35203 (local address 127.29.83.130:33409)
0924 20:18:44.380122 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.380133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.380268 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.380543 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.380546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.380563 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:44.380689 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.380694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.381483 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.381505 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.382466 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.382476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.383275 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.383278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.383368 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.383429 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.383519 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.383596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
I20260924 20:18:44.383814 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.379924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:49743 (local address 127.29.83.129:40809)
0924 20:18:44.380059 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.380065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.380360 (+ 295us) server_negotiation.cc:407] Connection header received
0924 20:18:44.380513 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.380516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.380569 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.380655 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:44.380934 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.381584 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.382749 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.383224 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.383369 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.383424 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.383612 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.383693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:18:44.383842 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.379804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40809 (local address 127.29.83.130:49743)
0924 20:18:44.380257 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.380268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:44.380405 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:44.380667 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:44.380670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:44.380686 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:44.380798 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.380803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.381697 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.381701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.382601 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:44.382610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.383369 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.383373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:44.383492 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.383509 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:18:44.383597 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:18:44.383667 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":25,"thread_start_us":68,"threads_started":1}
I20260924 20:18:44.384550 15759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd8078f3dda469a86d082acff4faff9" candidate_uuid: "7ad6217c930b43268566e358654d6efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6555e830745049f2898319a196cd6eff" is_pre_election: true
I20260924 20:18:44.384816 15759 raft_consensus.cc:2454] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad6217c930b43268566e358654d6efe in term 0.
I20260924 20:18:44.385313 15826 leader_election.cc:302] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6555e830745049f2898319a196cd6eff, 7ad6217c930b43268566e358654d6efe; no voters:
I20260924 20:18:44.385339 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd8078f3dda469a86d082acff4faff9" candidate_uuid: "7ad6217c930b43268566e358654d6efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f6382b8589408bbf49e1dcfd4d416c" is_pre_election: true
I20260924 20:18:44.385627 16021 raft_consensus.cc:2454] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad6217c930b43268566e358654d6efe in term 0.
I20260924 20:18:44.385669 16090 raft_consensus.cc:2781] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:44.385794 16090 raft_consensus.cc:482] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:44.385864 16090 raft_consensus.cc:3037] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:44.387081 16090 raft_consensus.cc:504] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.387518 16090 leader_election.cc:288] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [CANDIDATE]: Term 1 election: Requested vote from peers 42f6382b8589408bbf49e1dcfd4d416c (127.29.83.131:35203), 6555e830745049f2898319a196cd6eff (127.29.83.129:40809)
I20260924 20:18:44.387924 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd8078f3dda469a86d082acff4faff9" candidate_uuid: "7ad6217c930b43268566e358654d6efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f6382b8589408bbf49e1dcfd4d416c"
I20260924 20:18:44.388031 15759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd8078f3dda469a86d082acff4faff9" candidate_uuid: "7ad6217c930b43268566e358654d6efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6555e830745049f2898319a196cd6eff"
I20260924 20:18:44.388077 16021 raft_consensus.cc:3037] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:44.388164 15759 raft_consensus.cc:3037] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:44.389457 16021 raft_consensus.cc:2454] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad6217c930b43268566e358654d6efe in term 1.
I20260924 20:18:44.389457 15759 raft_consensus.cc:2454] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad6217c930b43268566e358654d6efe in term 1.
I20260924 20:18:44.389844 15824 leader_election.cc:302] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f6382b8589408bbf49e1dcfd4d416c, 7ad6217c930b43268566e358654d6efe; no voters:
I20260924 20:18:44.390014 16090 raft_consensus.cc:2781] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:44.390230 16090 raft_consensus.cc:686] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 1 LEADER]: Becoming Leader. State: Replica: 7ad6217c930b43268566e358654d6efe, State: Running, Role: LEADER
I20260924 20:18:44.390671 16090 consensus_queue.cc:237] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } }
I20260924 20:18:44.394230 15615 catalog_manager.cc:5495] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ad6217c930b43268566e358654d6efe (127.29.83.130). New cstate: current_term: 1 leader_uuid: "7ad6217c930b43268566e358654d6efe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad6217c930b43268566e358654d6efe" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 41179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:44.407485 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.403367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.29.83.130:41179)
0924 20:18:44.403536 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:44.403540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:44.403555 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:18:44.403721 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:44.403723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:44.403781 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:44.403898 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:44.404168 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.404769 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.405668 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:44.406055 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:44.406164 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:44.406486 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:44.407017 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:44.407162 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:44.407298 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 20:18:44.407338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:18:44.414235 15759 raft_consensus.cc:1261] T 5cd8078f3dda469a86d082acff4faff9 P 6555e830745049f2898319a196cd6eff [term 1 FOLLOWER]: Refusing update from remote peer 7ad6217c930b43268566e358654d6efe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:44.414235 16021 raft_consensus.cc:1261] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 1 FOLLOWER]: Refusing update from remote peer 7ad6217c930b43268566e358654d6efe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:44.415060 16095 consensus_queue.cc:1035] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f6382b8589408bbf49e1dcfd4d416c" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:44.415283 16090 consensus_queue.cc:1035] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6555e830745049f2898319a196cd6eff" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:44.423461 16098 mvcc.cc:205] Tried to move back new op lower bound from 7332991485592883200 to 7332991485507489792. Current Snapshot: MvccSnapshot[applied={T|T < 7332991485592883200}]
I20260924 20:18:44.432952 15615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:44.433141 15615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:44.435843 15615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89b2c534e49a4386b5b6748c1c24279a: Sending DeleteTablet for 3 replicas of tablet 5cd8078f3dda469a86d082acff4faff9
I20260924 20:18:44.436765 15739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd8078f3dda469a86d082acff4faff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:44 UTC) from {username='slave'} at 127.0.0.1:38704
I20260924 20:18:44.436791 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd8078f3dda469a86d082acff4faff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:44 UTC) from {username='slave'} at 127.0.0.1:60446
I20260924 20:18:44.437522 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15677
I20260924 20:18:44.437590 16108 tablet_replica.cc:330] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: stopping tablet replica
I20260924 20:18:44.437556 16001 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd8078f3dda469a86d082acff4faff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:44 UTC) from {username='slave'} at 127.0.0.1:54222
I20260924 20:18:44.437975 16108 raft_consensus.cc:2229] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:44.438366 16109 tablet_replica.cc:330] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: stopping tablet replica
I20260924 20:18:44.438441 16108 raft_consensus.cc:2258] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:44.438784 16109 raft_consensus.cc:2229] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:44.439131 16109 raft_consensus.cc:2258] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:44.440563 16108 ts_tablet_manager.cc:1905] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:44.440599 16109 ts_tablet_manager.cc:1905] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:44.443486 16109 ts_tablet_manager.cc:1918] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:44.443593 16109 log.cc:1196] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/5cd8078f3dda469a86d082acff4faff9
I20260924 20:18:44.443782 16108 ts_tablet_manager.cc:1918] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:44.443887 16108 log.cc:1196] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/5cd8078f3dda469a86d082acff4faff9
I20260924 20:18:44.443919 16109 ts_tablet_manager.cc:1939] T 5cd8078f3dda469a86d082acff4faff9 P 42f6382b8589408bbf49e1dcfd4d416c: Deleting consensus metadata
I20260924 20:18:44.444224 16108 ts_tablet_manager.cc:1939] T 5cd8078f3dda469a86d082acff4faff9 P 7ad6217c930b43268566e358654d6efe: Deleting consensus metadata
I20260924 20:18:44.445261 15602 catalog_manager.cc:4841] TS 42f6382b8589408bbf49e1dcfd4d416c (127.29.83.131:35203): tablet 5cd8078f3dda469a86d082acff4faff9 (table test-table [id=a313fb6386cb4cf39722b1b12b53c0f9]) successfully deleted
I20260924 20:18:44.445490 15602 catalog_manager.cc:4841] TS 7ad6217c930b43268566e358654d6efe (127.29.83.130:41179): tablet 5cd8078f3dda469a86d082acff4faff9 (table test-table [id=a313fb6386cb4cf39722b1b12b53c0f9]) successfully deleted
W20260924 20:18:44.448168 15604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:40809 (error 108)
I20260924 20:18:44.448380 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15808
I20260924 20:18:44.450059 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:44.449708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:40809 (local address 127.0.0.1:38720)
0924 20:18:44.449868 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:44.449951 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:40809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 20:18:44.450248 15604 catalog_manager.cc:4584] TS 6555e830745049f2898319a196cd6eff (127.29.83.129:40809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cd8078f3dda469a86d082acff4faff9: Network error: Client connection negotiation failed: client connection to 127.29.83.129:40809: connect: Connection refused (error 111)
I20260924 20:18:44.454280 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15939
I20260924 20:18:44.459683 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 15586
2026-09-24T20:18:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:44 dist-test-slave-52fr krb5kdc[16113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:44 dist-test-slave-52fr krb5kdc[16113](info): set up 2 sockets
Sep 24 20:18:44 dist-test-slave-52fr krb5kdc[16113](info): commencing operation
krb5kdc: starting...
W20260924 20:18:45.513891 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:45 dist-test-slave-52fr krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:18:45.645383 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35305
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:35305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:45.760751 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:45.761169 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:45.761219 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:45.765863 16126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:45.765977 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:45.766003 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:45.766022 16126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:45.766083 16126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:45.772465 16126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:35305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:35305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:45.773697 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:45.774876 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:45.784950 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:45.784960 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:45.785276 16126 server_base.cc:1034] running on GCE node
W20260924 20:18:45.785034 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:45.786018 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:45.787360 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:45.788581 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790281125788554 us; error 39 us; skew 500 ppm
Sep 24 20:18:45 dist-test-slave-52fr krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:45.797202 16126 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:18:45.798341 16126 webserver.cc:460] Webserver started at http://127.29.83.190:44551/ using document root <none> and password file <none>
I20260924 20:18:45.799208 16126 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:45.799310 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:45.799574 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:45.802649 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "9dec3d42019b45d08db8609b4e68a895"
format_stamp: "Formatted at 2026-09-24 20:18:45 on dist-test-slave-52fr"
server_key: "fbe506a53c035560df41abc828e20931"
server_key_iv: "ba9ed9dd6c8f2f391ac17e132416407a"
server_key_version: "encryptionkey@0"
I20260924 20:18:45.803328 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "9dec3d42019b45d08db8609b4e68a895"
format_stamp: "Formatted at 2026-09-24 20:18:45 on dist-test-slave-52fr"
server_key: "fbe506a53c035560df41abc828e20931"
server_key_iv: "ba9ed9dd6c8f2f391ac17e132416407a"
server_key_version: "encryptionkey@0"
I20260924 20:18:45.807600 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:45.810621 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:45.812238 16126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:45.812395 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "9dec3d42019b45d08db8609b4e68a895"
format_stamp: "Formatted at 2026-09-24 20:18:45 on dist-test-slave-52fr"
server_key: "fbe506a53c035560df41abc828e20931"
server_key_iv: "ba9ed9dd6c8f2f391ac17e132416407a"
server_key_version: "encryptionkey@0"
I20260924 20:18:45.812536 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:45.822822 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:45.826751 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:45.826979 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:45.835376 16126 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:35305
I20260924 20:18:45.835414 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:35305 every 8 connection(s)
I20260924 20:18:45.836555 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:45.839849 16194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:45.842639 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16126
I20260924 20:18:45.842780 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:45.843144 30030 external_mini_cluster.cc:1428] Setting key d1cf2c8f16297f4af56b81e202c8231b
I20260924 20:18:45.846446 16194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: Bootstrap starting.
I20260924 20:18:45.848872 16194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:45.849699 16194 log.cc:824] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:45.851938 16194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: No bootstrap required, opened a new log
Sep 24 20:18:45 dist-test-slave-52fr krb5kdc[16113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:45.855773 16194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } }
I20260924 20:18:45.856029 16194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:45.856065 16194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec3d42019b45d08db8609b4e68a895, State: Initialized, Role: FOLLOWER
I20260924 20:18:45.856557 16194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } }
I20260924 20:18:45.856721 16194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:45.856796 16194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:45.856931 16194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:45.858028 16194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } }
I20260924 20:18:45.858414 16194 leader_election.cc:302] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dec3d42019b45d08db8609b4e68a895; no voters:
I20260924 20:18:45.858794 16194 leader_election.cc:288] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:45.858909 16199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:45.859148 16199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [term 1 LEADER]: Becoming Leader. State: Replica: 9dec3d42019b45d08db8609b4e68a895, State: Running, Role: LEADER
I20260924 20:18:45.859505 16199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } }
I20260924 20:18:45.860191 16194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:45.860559 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:45.844668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.29.83.190:35305)
0924 20:18:45.845123 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:45.845140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:45.845179 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:18:45.845924 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:45.845947 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:45.846325 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:45.846700 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:45.847589 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:45.848644 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:45.849647 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:45.850286 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:45.850480 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:45.853850 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:45.853858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:45.853874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:45.853908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:45.856585 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:45.857160 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:45.857166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:45.857172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:45.857302 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:45.857816 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:45.857823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:45.857824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:45.858388 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:45.858608 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:45.859089 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 20:18:45.859481 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":148,"threads_started":1}
I20260924 20:18:45.861367 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dec3d42019b45d08db8609b4e68a895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } } }
I20260924 20:18:45.861413 16201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dec3d42019b45d08db8609b4e68a895. Latest consensus state: current_term: 1 leader_uuid: "9dec3d42019b45d08db8609b4e68a895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec3d42019b45d08db8609b4e68a895" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 35305 } } }
I20260924 20:18:45.861502 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:45.861505 16201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:45.861882 16203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:45.865123 16203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:45.871116 16203 catalog_manager.cc:1347] Generated new cluster ID: ed69bcf5863e486cb4c0be164c93f75e
I20260924 20:18:45.871208 16203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:45.892030 16203 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:45.893076 16203 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:45.905002 16203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: Generated new TSK 0
I20260924 20:18:45.905984 16203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.83.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
Entry for principal kudu/127.29.83.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab.
I20260924 20:18:45.955011 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:46.080719 16220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:46.081086 16220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:46.081175 16220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:46.085538 16220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:46.085647 16220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:46.085783 16220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:46.091462 16220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.129.keytab
--principal=kudu/127.29.83.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:46.092619 16220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:46.093730 16220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:46.104432 16225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:46.104420 16228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.104494 16220 server_base.cc:1034] running on GCE node
W20260924 20:18:46.104421 16226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.105229 16220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:46.106078 16220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:46.107303 16220 hybrid_clock.cc:648] HybridClock initialized: now 1790281126107271 us; error 46 us; skew 500 ppm
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:46.110656 16220 init.cc:377] Logged in from keytab as kudu/127.29.83.129@KRBTEST.COM (short username kudu)
I20260924 20:18:46.111718 16220 webserver.cc:460] Webserver started at http://127.29.83.129:33873/ using document root <none> and password file <none>
I20260924 20:18:46.112339 16220 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:46.112432 16220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:46.112670 16220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:46.114878 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "7d87735c6c044509b4eab5dfe182a982"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "84d1f8d5657dfd6008e1b4fb7c8afb8a"
server_key_iv: "b3e6ade01c8521412b82304ccf6ccc5d"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.115504 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "7d87735c6c044509b4eab5dfe182a982"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "84d1f8d5657dfd6008e1b4fb7c8afb8a"
server_key_iv: "b3e6ade01c8521412b82304ccf6ccc5d"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.119580 16220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:46.122164 16235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.123519 16220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:18:46.123675 16220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "7d87735c6c044509b4eab5dfe182a982"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "84d1f8d5657dfd6008e1b4fb7c8afb8a"
server_key_iv: "b3e6ade01c8521412b82304ccf6ccc5d"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.123804 16220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:46.148397 16220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:46.152004 16220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:46.152225 16220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:46.152884 16220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:46.153738 16220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:46.153839 16220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.153916 16220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:46.153966 16220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.163231 16220 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:45325
I20260924 20:18:46.163256 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:45325 every 8 connection(s)
I20260924 20:18:46.164207 16220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:46.172629 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16220
I20260924 20:18:46.172793 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:18:46.173170 30030 external_mini_cluster.cc:1428] Setting key aefbd2ff4f57d74a22cb9ed156a0d1a0
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.129@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:46.179126 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.166258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:51587 (local address 127.29.83.190:35305)
0924 20:18:46.166446 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.166451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.167370 (+ 919us) server_negotiation.cc:407] Connection header received
0924 20:18:46.168698 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.168701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.168757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.168870 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:46.170145 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.170726 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.171908 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.172145 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.172236 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.175325 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:46.175328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:46.175331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:46.175355 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:46.176908 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.177445 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.177450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.177452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.177539 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.177964 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.177969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.177971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.178170 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:46.178281 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.178813 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.178957 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 20:18:46.180032 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.166736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35305 (local address 127.29.83.129:51587)
0924 20:18:46.167212 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.167256 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.168480 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.169048 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.169055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.169447 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:46.169978 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.169993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.170857 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.170859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.171788 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.171797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.172286 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.172289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.172550 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.173191 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:46.173216 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:46.175142 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:46.177038 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.177044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.177062 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.177340 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.177641 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.177645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.177647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.177869 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.178280 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:46.178289 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:46.178566 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.178756 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.179116 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260924 20:18:46.181389 16349 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35305
I20260924 20:18:46.181663 16349 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:46.182317 16349 heartbeater.cc:511] Master 127.29.83.190:35305 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.83.130@KRBTEST.COM; defaulting to no policy
I20260924 20:18:46.184042 16158 ts_manager.cc:194] Registered new tserver with Master: 7d87735c6c044509b4eab5dfe182a982 (127.29.83.129:45325)
I20260924 20:18:46.185231 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.129@KRBTEST.COM'} at 127.29.83.129:51587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
Entry for principal kudu/127.29.83.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab.
I20260924 20:18:46.204733 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:46.316201 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:46.316488 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:46.316565 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:46.320780 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:46.320847 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:46.320953 16354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:46.326428 16354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.130.keytab
--principal=kudu/127.29.83.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:46.327584 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:46.328697 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:46.338878 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:46.338878 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.339005 16354 server_base.cc:1034] running on GCE node
W20260924 20:18:46.338937 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.339521 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:46.340193 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:46.341387 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790281126341345 us; error 55 us; skew 500 ppm
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:46.344197 16354 init.cc:377] Logged in from keytab as kudu/127.29.83.130@KRBTEST.COM (short username kudu)
I20260924 20:18:46.345137 16354 webserver.cc:460] Webserver started at http://127.29.83.130:43825/ using document root <none> and password file <none>
I20260924 20:18:46.345719 16354 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:46.345837 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:46.346077 16354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:46.348141 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "1a93f253d0354b3faec8aacc088e8edd"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "c8bebd556746d4bfb5b9f776e411f7a4"
server_key_iv: "602222c64ed47d39468f0f372bd9838e"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.348728 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "1a93f253d0354b3faec8aacc088e8edd"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "c8bebd556746d4bfb5b9f776e411f7a4"
server_key_iv: "602222c64ed47d39468f0f372bd9838e"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.352665 16354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:18:46.355253 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.356565 16354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:18:46.356704 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "1a93f253d0354b3faec8aacc088e8edd"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "c8bebd556746d4bfb5b9f776e411f7a4"
server_key_iv: "602222c64ed47d39468f0f372bd9838e"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.356818 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:46.369495 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:46.372491 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:46.372692 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:46.373275 16354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:46.374150 16354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:46.374221 16354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.374293 16354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:46.374344 16354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.383397 16354 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:34095
I20260924 20:18:46.383419 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:34095 every 8 connection(s)
I20260924 20:18:46.384294 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:46.391635 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16354
I20260924 20:18:46.391782 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:46.392099 30030 external_mini_cluster.cc:1428] Setting key e294977f4d6cfe959f93dd5cce3bdd8e
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.130@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:46.398512 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.386130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:40209 (local address 127.29.83.190:35305)
0924 20:18:46.386274 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.386278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.387151 (+ 873us) server_negotiation.cc:407] Connection header received
0924 20:18:46.388384 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.388387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.388439 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.388511 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:46.389581 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.390185 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.391327 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.391688 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.391813 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.394573 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:46.394580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:46.394586 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:46.394621 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:46.396199 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.396825 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.396828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.396829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.396903 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.397283 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.397285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.397286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.397476 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:46.397550 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.398265 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.398372 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:18:46.399868 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.386561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35305 (local address 127.29.83.130:40209)
0924 20:18:46.386993 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.387035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.388192 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.388640 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.388647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.388979 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:46.389457 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.389470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.390303 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.390306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.391218 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.391227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.391893 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.391897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.392046 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.392722 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:46.392743 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:46.394407 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:46.396319 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.396327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.396346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.396721 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.396991 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.396997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.397002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.397192 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.397585 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:46.397595 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:46.397989 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.398226 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.398610 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260924 20:18:46.401158 16483 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35305
I20260924 20:18:46.401435 16483 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:46.402000 16483 heartbeater.cc:511] Master 127.29.83.190:35305 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.83.131@KRBTEST.COM; defaulting to no policy
I20260924 20:18:46.403115 16158 ts_manager.cc:194] Registered new tserver with Master: 1a93f253d0354b3faec8aacc088e8edd (127.29.83.130:34095)
I20260924 20:18:46.403712 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.130@KRBTEST.COM'} at 127.29.83.130:40209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
Entry for principal kudu/127.29.83.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab.
I20260924 20:18:46.420105 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:46.551282 16488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:46.551611 16488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:46.551688 16488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:46.555788 16488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:46.555924 16488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:46.556026 16488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:46.561657 16488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.131.keytab
--principal=kudu/127.29.83.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:35305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:46.562906 16488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:46.564034 16488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:46.574086 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:46.574100 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.574252 16488 server_base.cc:1034] running on GCE node
W20260924 20:18:46.574424 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:46.574851 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:46.575469 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:46.576627 16488 hybrid_clock.cc:648] HybridClock initialized: now 1790281126576596 us; error 41 us; skew 500 ppm
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:46.579490 16488 init.cc:377] Logged in from keytab as kudu/127.29.83.131@KRBTEST.COM (short username kudu)
I20260924 20:18:46.580453 16488 webserver.cc:460] Webserver started at http://127.29.83.131:35769/ using document root <none> and password file <none>
I20260924 20:18:46.581063 16488 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:46.581151 16488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:46.581394 16488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:46.583494 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "a449c23dcada4d4a979f075e6feec96d"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "1a7818b2ee9f7376011164868d4a18cb"
server_key_iv: "15f73f285234ecd9314478d599f56638"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.584048 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "a449c23dcada4d4a979f075e6feec96d"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "1a7818b2ee9f7376011164868d4a18cb"
server_key_iv: "15f73f285234ecd9314478d599f56638"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.587913 16488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:18:46.590507 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.591796 16488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:18:46.591933 16488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "a449c23dcada4d4a979f075e6feec96d"
format_stamp: "Formatted at 2026-09-24 20:18:46 on dist-test-slave-52fr"
server_key: "1a7818b2ee9f7376011164868d4a18cb"
server_key_iv: "15f73f285234ecd9314478d599f56638"
server_key_version: "encryptionkey@0"
I20260924 20:18:46.592048 16488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:46.619405 16488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:46.623036 16488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:46.623265 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:46.624085 16488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:46.625129 16488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:46.625209 16488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.625296 16488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:46.625350 16488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:46.634541 16488 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:43831
I20260924 20:18:46.634565 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:43831 every 8 connection(s)
I20260924 20:18:46.635473 16488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:46.637472 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16488
I20260924 20:18:46.637640 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:18:46.638079 30030 external_mini_cluster.cc:1428] Setting key 30523298c4b5595c2b3b4eaca76032e1
Sep 24 20:18:46 dist-test-slave-52fr krb5kdc[16113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281126, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.131@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:18:46.650053 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.637449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:56019 (local address 127.29.83.190:35305)
0924 20:18:46.637639 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.637645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.638694 (+ 1049us) server_negotiation.cc:407] Connection header received
0924 20:18:46.640314 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.640319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.640369 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.640448 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:46.641580 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.642202 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.643454 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.643673 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.643767 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.646727 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:46.646730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:46.646732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:46.646758 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:46.648191 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.648673 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.648676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.648677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.648752 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.649042 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.649044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.649045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.649220 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:46.649294 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.649803 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.649928 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:18:46.651031 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.637954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:35305 (local address 127.29.83.131:56019)
0924 20:18:46.638533 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.638579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.640050 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.640593 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.640599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.640951 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:46.641447 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.641459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.642336 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.642339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.643311 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.643320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.643864 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.643868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.644030 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.644838 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:18:46.644857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:46.646549 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:46.648297 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.648302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.648316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.648581 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.648831 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:18:46.648834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:18:46.648835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:18:46.648965 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:18:46.649305 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:46.649310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:46.649554 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.649708 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.650058 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":152,"threads_started":1}
I20260924 20:18:46.652215 16617 heartbeater.cc:347] Connected to a master server at 127.29.83.190:35305
I20260924 20:18:46.652433 16617 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:46.652860 16617 heartbeater.cc:511] Master 127.29.83.190:35305 requested a full tablet report, sending...
I20260924 20:18:46.653854 16158 ts_manager.cc:194] Registered new tserver with Master: a449c23dcada4d4a979f075e6feec96d (127.29.83.131:43831)
I20260924 20:18:46.654446 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.83.131@KRBTEST.COM'} at 127.29.83.131:56019
I20260924 20:18:46.664274 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:46.674034 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.666335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.29.83.190:35305)
0924 20:18:46.666498 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.666503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.666653 (+ 150us) server_negotiation.cc:407] Connection header received
0924 20:18:46.666807 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.666810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.666860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.666942 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:46.667667 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.668195 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.669046 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.669263 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.669371 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.670871 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:46.670876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:46.670879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:46.670911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:46.672150 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.672610 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.672616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.672618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.672719 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:46.673020 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:46.673025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:46.673027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:46.673296 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:46.673451 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.673707 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.673849 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:18:46.676761 16158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:46.679944 16158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:46.695268 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.689515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:43831 (local address 127.0.0.1:43320)
0924 20:18:46.689894 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.689908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.689992 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.691139 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.691144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.691165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:46.691252 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.691259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.692717 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.692722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.693600 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.693609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694677 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.694680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.694847 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.694886 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.694968 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.695061 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":19,"thread_start_us":114,"threads_started":1}
I20260924 20:18:46.695341 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.689989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.29.83.130:34095)
0924 20:18:46.690493 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.690502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.690544 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:18:46.690620 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.690625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.690802 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.690944 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:46.691183 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.692315 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.693624 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694532 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.694725 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.694872 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.695095 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.695205 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":132,"threads_started":1}
I20260924 20:18:46.695324 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.689641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:34095 (local address 127.0.0.1:44548)
0924 20:18:46.690165 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.690178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.690264 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.690947 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.690950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.690970 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:46.691059 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.691065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.692460 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.692464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.693505 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.693515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694677 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.694680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.694831 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.694888 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.695064 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.695165 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":87,"thread_start_us":119,"threads_started":1}
I20260924 20:18:46.696120 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.688923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45325 (local address 127.0.0.1:43052)
0924 20:18:46.689285 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.689376 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.689528 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.690367 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.690373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.690399 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:46.690545 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.690552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.691677 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.691682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.693817 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.693834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694658 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.694662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.695802 (+ 1140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.695845 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.695916 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.695984 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1088,"thread_start_us":75,"threads_started":1}
I20260924 20:18:46.696341 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.690371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.29.83.131:43831)
0924 20:18:46.690693 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.690703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.690730 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:18:46.690797 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.690801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.690959 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.691400 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:46.691527 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.692599 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.693730 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694532 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.695758 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.695931 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.695977 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.696088 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
I20260924 20:18:46.696549 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.689079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.29.83.129:45325)
0924 20:18:46.689373 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.689381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.689682 (+ 301us) server_negotiation.cc:407] Connection header received
0924 20:18:46.689768 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.689794 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.690017 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.690174 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:46.690685 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.691567 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.693961 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.694530 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.694659 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.694818 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.696287 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.696401 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260924 20:18:46.698889 16283 tablet_service.cc:1467] Processing CreateTablet for tablet 82d34066c46e4a20aae38a488f422808 (DEFAULT_TABLE table=test-table [id=f00c30f96be3499183374a2c13ed817b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:46.698877 16417 tablet_service.cc:1467] Processing CreateTablet for tablet 82d34066c46e4a20aae38a488f422808 (DEFAULT_TABLE table=test-table [id=f00c30f96be3499183374a2c13ed817b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:46.698879 16551 tablet_service.cc:1467] Processing CreateTablet for tablet 82d34066c46e4a20aae38a488f422808 (DEFAULT_TABLE table=test-table [id=f00c30f96be3499183374a2c13ed817b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:46.699842 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d34066c46e4a20aae38a488f422808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:46.699863 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d34066c46e4a20aae38a488f422808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:46.699863 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d34066c46e4a20aae38a488f422808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:46.706213 16634 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: Bootstrap starting.
I20260924 20:18:46.707738 16636 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Bootstrap starting.
I20260924 20:18:46.707914 16635 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Bootstrap starting.
I20260924 20:18:46.708665 16634 tablet_bootstrap.cc:654] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:46.709435 16634 log.cc:824] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:46.710065 16635 tablet_bootstrap.cc:654] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:46.710065 16636 tablet_bootstrap.cc:654] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:46.711423 16635 log.cc:824] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:46.711423 16636 log.cc:824] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:46.711742 16634 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: No bootstrap required, opened a new log
I20260924 20:18:46.711961 16634 ts_tablet_manager.cc:1397] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 20:18:46.713493 16636 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: No bootstrap required, opened a new log
I20260924 20:18:46.713495 16635 tablet_bootstrap.cc:492] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: No bootstrap required, opened a new log
I20260924 20:18:46.713683 16635 ts_tablet_manager.cc:1397] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:18:46.713683 16636 ts_tablet_manager.cc:1397] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:18:46.715586 16634 raft_consensus.cc:348] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.715896 16634 raft_consensus.cc:374] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:46.715960 16634 raft_consensus.cc:729] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d87735c6c044509b4eab5dfe182a982, State: Initialized, Role: FOLLOWER
I20260924 20:18:46.716512 16634 consensus_queue.cc:260] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.717540 16634 ts_tablet_manager.cc:1428] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 20:18:46.717569 16635 raft_consensus.cc:348] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.717852 16635 raft_consensus.cc:374] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:46.717903 16635 raft_consensus.cc:729] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a449c23dcada4d4a979f075e6feec96d, State: Initialized, Role: FOLLOWER
I20260924 20:18:46.717932 16349 heartbeater.cc:503] Master 127.29.83.190:35305 was elected leader, sending a full tablet report...
I20260924 20:18:46.718357 16635 consensus_queue.cc:260] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.717569 16636 raft_consensus.cc:348] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.718968 16636 raft_consensus.cc:374] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:46.719020 16636 raft_consensus.cc:729] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a93f253d0354b3faec8aacc088e8edd, State: Initialized, Role: FOLLOWER
I20260924 20:18:46.719012 16617 heartbeater.cc:503] Master 127.29.83.190:35305 was elected leader, sending a full tablet report...
I20260924 20:18:46.719373 16635 ts_tablet_manager.cc:1428] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:18:46.719429 16636 consensus_queue.cc:260] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.720042 16483 heartbeater.cc:503] Master 127.29.83.190:35305 was elected leader, sending a full tablet report...
I20260924 20:18:46.721112 16636 ts_tablet_manager.cc:1428] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:18:46.770053 16642 raft_consensus.cc:482] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:46.770354 16642 raft_consensus.cc:504] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.771569 16642 leader_election.cc:288] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d87735c6c044509b4eab5dfe182a982 (127.29.83.129:45325), a449c23dcada4d4a979f075e6feec96d (127.29.83.131:43831)
I20260924 20:18:46.776693 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.771902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:47565 (local address 127.29.83.129:45325)
0924 20:18:46.772024 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.772029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.772046 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:18:46.772105 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.772108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.772167 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.772250 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:46.772589 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.773398 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775214 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.775849 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775967 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.776017 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.776487 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.776565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:18:46.776697 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.771753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45325 (local address 127.29.83.130:47565)
0924 20:18:46.771899 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.771916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.772017 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.772290 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.772296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.772317 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:46.772448 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.772454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.773508 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.773511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.775071 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.775082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.775971 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.776210 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.776346 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.776428 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.776533 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":197}
I20260924 20:18:46.776693 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.772030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:40545 (local address 127.29.83.131:43831)
0924 20:18:46.772149 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.772154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.772336 (+ 182us) server_negotiation.cc:407] Connection header received
0924 20:18:46.772403 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.772405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.772446 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.772503 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:46.772751 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.773470 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775214 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.775819 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775967 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.776017 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.776487 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.776565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:18:46.776711 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.771847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:43831 (local address 127.29.83.130:40545)
0924 20:18:46.772212 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.772241 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:46.772325 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:46.772537 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:46.772539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:46.772558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:46.772649 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.772653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.773570 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.773573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.775070 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:46.775082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.775971 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.775975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:46.776209 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.776346 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 20:18:46.776428 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:18:46.776531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":303,"thread_start_us":136,"threads_started":1}
I20260924 20:18:46.777400 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d34066c46e4a20aae38a488f422808" candidate_uuid: "1a93f253d0354b3faec8aacc088e8edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d87735c6c044509b4eab5dfe182a982" is_pre_election: true
I20260924 20:18:46.777376 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d34066c46e4a20aae38a488f422808" candidate_uuid: "1a93f253d0354b3faec8aacc088e8edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a449c23dcada4d4a979f075e6feec96d" is_pre_election: true
I20260924 20:18:46.777745 16303 raft_consensus.cc:2454] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a93f253d0354b3faec8aacc088e8edd in term 0.
I20260924 20:18:46.777745 16571 raft_consensus.cc:2454] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a93f253d0354b3faec8aacc088e8edd in term 0.
I20260924 20:18:46.778344 16370 leader_election.cc:302] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a93f253d0354b3faec8aacc088e8edd, 7d87735c6c044509b4eab5dfe182a982, a449c23dcada4d4a979f075e6feec96d; no voters:
I20260924 20:18:46.778690 16642 raft_consensus.cc:2781] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:46.778801 16642 raft_consensus.cc:482] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:46.778882 16642 raft_consensus.cc:3037] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:46.779943 16642 raft_consensus.cc:504] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.780297 16642 leader_election.cc:288] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [CANDIDATE]: Term 1 election: Requested vote from peers 7d87735c6c044509b4eab5dfe182a982 (127.29.83.129:45325), a449c23dcada4d4a979f075e6feec96d (127.29.83.131:43831)
I20260924 20:18:46.780673 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d34066c46e4a20aae38a488f422808" candidate_uuid: "1a93f253d0354b3faec8aacc088e8edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a449c23dcada4d4a979f075e6feec96d"
I20260924 20:18:46.780687 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d34066c46e4a20aae38a488f422808" candidate_uuid: "1a93f253d0354b3faec8aacc088e8edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d87735c6c044509b4eab5dfe182a982"
I20260924 20:18:46.780777 16571 raft_consensus.cc:3037] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:46.780822 16303 raft_consensus.cc:3037] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:46.781873 16571 raft_consensus.cc:2454] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a93f253d0354b3faec8aacc088e8edd in term 1.
I20260924 20:18:46.782181 16370 leader_election.cc:302] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a93f253d0354b3faec8aacc088e8edd, a449c23dcada4d4a979f075e6feec96d; no voters:
I20260924 20:18:46.782361 16642 raft_consensus.cc:2781] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:46.782356 16303 raft_consensus.cc:2454] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a93f253d0354b3faec8aacc088e8edd in term 1.
I20260924 20:18:46.782588 16642 raft_consensus.cc:686] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 1 LEADER]: Becoming Leader. State: Replica: 1a93f253d0354b3faec8aacc088e8edd, State: Running, Role: LEADER
I20260924 20:18:46.782954 16642 consensus_queue.cc:237] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } }
I20260924 20:18:46.786067 16158 catalog_manager.cc:5495] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a93f253d0354b3faec8aacc088e8edd (127.29.83.130). New cstate: current_term: 1 leader_uuid: "1a93f253d0354b3faec8aacc088e8edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a93f253d0354b3faec8aacc088e8edd" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 34095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:46.818564 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.814799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.29.83.130:34095)
0924 20:18:46.814939 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:46.814944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:46.814972 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:18:46.815066 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:46.815068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:46.815124 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:46.815245 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:46.815538 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.816071 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.816980 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:46.817204 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:46.817310 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:46.817645 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:46.818128 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:46.818270 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:46.818334 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 20:18:46.818393 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:18:46.824472 16571 raft_consensus.cc:1261] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 1 FOLLOWER]: Refusing update from remote peer 1a93f253d0354b3faec8aacc088e8edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:46.824472 16303 raft_consensus.cc:1261] T 82d34066c46e4a20aae38a488f422808 P 7d87735c6c044509b4eab5dfe182a982 [term 1 FOLLOWER]: Refusing update from remote peer 1a93f253d0354b3faec8aacc088e8edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:46.825172 16644 consensus_queue.cc:1035] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d87735c6c044509b4eab5dfe182a982" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:46.825366 16642 consensus_queue.cc:1035] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a449c23dcada4d4a979f075e6feec96d" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:46.833675 16648 mvcc.cc:205] Tried to move back new op lower bound from 7332991495465668608 to 7332991495305375744. Current Snapshot: MvccSnapshot[applied={T|T < 7332991495465668608}]
I20260924 20:18:46.849152 16156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:46.849346 16156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:46.851850 16156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dec3d42019b45d08db8609b4e68a895: Sending DeleteTablet for 3 replicas of tablet 82d34066c46e4a20aae38a488f422808
I20260924 20:18:46.852814 16417 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d34066c46e4a20aae38a488f422808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:46 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:44548
I20260924 20:18:46.852814 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d34066c46e4a20aae38a488f422808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:46 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:43052
I20260924 20:18:46.852846 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d34066c46e4a20aae38a488f422808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:46 UTC) from {username='kudu', principal='kudu/127.29.83.190@KRBTEST.COM'} at 127.0.0.1:43320
I20260924 20:18:46.853432 16658 tablet_replica.cc:330] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: stopping tablet replica
I20260924 20:18:46.853489 16659 tablet_replica.cc:330] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: stopping tablet replica
I20260924 20:18:46.853861 16658 raft_consensus.cc:2229] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:46.853874 16659 raft_consensus.cc:2229] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:46.854084 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16220
I20260924 20:18:46.854293 16659 raft_consensus.cc:2258] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:46.854432 16658 raft_consensus.cc:2258] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:46.856194 16658 ts_tablet_manager.cc:1905] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:46.856263 16659 ts_tablet_manager.cc:1905] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:46.859213 16659 ts_tablet_manager.cc:1918] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:46.859331 16659 log.cc:1196] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/82d34066c46e4a20aae38a488f422808
I20260924 20:18:46.859680 16659 ts_tablet_manager.cc:1939] T 82d34066c46e4a20aae38a488f422808 P a449c23dcada4d4a979f075e6feec96d: Deleting consensus metadata
I20260924 20:18:46.859674 16658 ts_tablet_manager.cc:1918] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:46.859934 16658 log.cc:1196] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280784619584-30030-0/minicluster-data/ts-1/wal/wals/82d34066c46e4a20aae38a488f422808
I20260924 20:18:46.860356 16658 ts_tablet_manager.cc:1939] T 82d34066c46e4a20aae38a488f422808 P 1a93f253d0354b3faec8aacc088e8edd: Deleting consensus metadata
I20260924 20:18:46.861109 16142 catalog_manager.cc:4841] TS a449c23dcada4d4a979f075e6feec96d (127.29.83.131:43831): tablet 82d34066c46e4a20aae38a488f422808 (table test-table [id=f00c30f96be3499183374a2c13ed817b]) successfully deleted
I20260924 20:18:46.861661 16144 catalog_manager.cc:4841] TS 1a93f253d0354b3faec8aacc088e8edd (127.29.83.130:34095): tablet 82d34066c46e4a20aae38a488f422808 (table test-table [id=f00c30f96be3499183374a2c13ed817b]) successfully deleted
W20260924 20:18:46.864454 16144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:45325 (error 108)
I20260924 20:18:46.865283 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16354
I20260924 20:18:46.865545 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:46.865216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:45325 (local address 127.0.0.1:43060)
0924 20:18:46.865390 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:46.865464 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 20:18:46.866000 16144 catalog_manager.cc:4584] TS 7d87735c6c044509b4eab5dfe182a982 (127.29.83.129:45325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82d34066c46e4a20aae38a488f422808: Network error: Client connection negotiation failed: client connection to 127.29.83.129:45325: connect: Connection refused (error 111)
I20260924 20:18:46.871173 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16488
I20260924 20:18:46.876609 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16126
2026-09-24T20:18:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2417 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T20:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:46Z Disabled control of system clock
I20260924 20:18:46.907622 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45509
--webserver_interface=127.29.83.190
--webserver_port=0
--builtin_ntp_servers=127.29.83.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:47.021191 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:47.021487 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:47.021524 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:47.025622 16664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:47.025702 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:47.025718 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:47.025736 16664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:47.025753 16664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:47.030689 16664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:47.031821 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:47.033212 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:47.041841 16669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.041836 16670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.041836 16672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:47.042281 16664 server_base.cc:1034] running on GCE node
I20260924 20:18:47.042891 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:47.044009 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:47.045225 16664 hybrid_clock.cc:648] HybridClock initialized: now 1790281127045204 us; error 39 us; skew 500 ppm
I20260924 20:18:47.047403 16664 webserver.cc:460] Webserver started at http://127.29.83.190:43209/ using document root <none> and password file <none>
I20260924 20:18:47.047976 16664 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:47.048036 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:47.048204 16664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:47.050359 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "bda048b16c944923a58eab13b9e39372"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "f185bf8b4ae45b358a59839f715cb988"
server_key_iv: "48af8a7758f91b0cb0b71df82dd1c930"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.050884 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "bda048b16c944923a58eab13b9e39372"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "f185bf8b4ae45b358a59839f715cb988"
server_key_iv: "48af8a7758f91b0cb0b71df82dd1c930"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.054613 16664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:47.057066 16678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.058480 16664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:47.058635 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "bda048b16c944923a58eab13b9e39372"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "f185bf8b4ae45b358a59839f715cb988"
server_key_iv: "48af8a7758f91b0cb0b71df82dd1c930"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.058753 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:47.074720 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:47.075527 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:47.075752 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:47.082698 16664 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:45509
I20260924 20:18:47.082716 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:45509 every 8 connection(s)
I20260924 20:18:47.083668 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:47.087026 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:47.092785 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: Bootstrap starting.
I20260924 20:18:47.093892 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16664
I20260924 20:18:47.094066 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:47.094384 30030 external_mini_cluster.cc:1428] Setting key dbaf95a160ce711fa073a9b55b7693a2
I20260924 20:18:47.096081 16731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:47.096911 16731 log.cc:824] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:47.099066 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: No bootstrap required, opened a new log
I20260924 20:18:47.102821 16731 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } }
I20260924 20:18:47.103111 16731 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:47.103168 16731 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda048b16c944923a58eab13b9e39372, State: Initialized, Role: FOLLOWER
I20260924 20:18:47.103649 16731 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } }
I20260924 20:18:47.103809 16731 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:47.103900 16731 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:47.104028 16731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:47.104357 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.095594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.29.83.190:45509)
0924 20:18:47.096383 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.096400 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.096438 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:18:47.097285 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.097307 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.097672 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.098095 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:47.098629 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.099749 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.100694 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.101209 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.101399 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.102375 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:47.102380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:47.102396 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:47.102806 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:47.102865 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:47.102885 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:47.103000 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.103118 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.103443 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260924 20:18:47.105265 16731 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } }
I20260924 20:18:47.105620 16731 leader_election.cc:302] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda048b16c944923a58eab13b9e39372; no voters:
I20260924 20:18:47.105980 16731 leader_election.cc:288] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:47.106098 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:47.106329 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [term 1 LEADER]: Becoming Leader. State: Replica: bda048b16c944923a58eab13b9e39372, State: Running, Role: LEADER
I20260924 20:18:47.106647 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } }
I20260924 20:18:47.107401 16731 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:47.108410 16737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda048b16c944923a58eab13b9e39372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } } }
I20260924 20:18:47.108394 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda048b16c944923a58eab13b9e39372. Latest consensus state: current_term: 1 leader_uuid: "bda048b16c944923a58eab13b9e39372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda048b16c944923a58eab13b9e39372" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45509 } } }
I20260924 20:18:47.108547 16737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:47.108572 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:47.108930 16745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:47.113304 16745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:47.119500 16745 catalog_manager.cc:1347] Generated new cluster ID: cfe06db30cbb47c1b2af433e16fe0fa6
I20260924 20:18:47.119601 16745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:47.128911 16745 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:47.130021 16745 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:47.141535 16745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: Generated new TSK 0
I20260924 20:18:47.142414 16745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:47.147763 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.129:0
--local_ip_for_outbound_sockets=127.29.83.129
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--builtin_ntp_servers=127.29.83.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:18:47.316895 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:47.317167 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:47.317207 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:47.321403 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:47.321466 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:47.321522 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.129
I20260924 20:18:47.326535 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.83.129
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.129
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:47.327920 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:47.329355 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:47.343012 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.343037 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.343003 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:47.343641 16755 server_base.cc:1034] running on GCE node
I20260924 20:18:47.344024 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:47.344681 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:47.345849 16755 hybrid_clock.cc:648] HybridClock initialized: now 1790281127345804 us; error 53 us; skew 500 ppm
I20260924 20:18:47.347854 16755 webserver.cc:460] Webserver started at http://127.29.83.129:42367/ using document root <none> and password file <none>
I20260924 20:18:47.348469 16755 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:47.348569 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:47.348809 16755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:47.350919 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data/instance:
uuid: "09a0f0de1546450199f75028a0da0ded"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "dada36403bc6929054437374683ded7b"
server_key_iv: "a4f954a1163dd39ed6bcd231a3d3b861"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.351485 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance:
uuid: "09a0f0de1546450199f75028a0da0ded"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "dada36403bc6929054437374683ded7b"
server_key_iv: "a4f954a1163dd39ed6bcd231a3d3b861"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.355360 16755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:47.357854 16769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.359127 16755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:18:47.359242 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal
uuid: "09a0f0de1546450199f75028a0da0ded"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "dada36403bc6929054437374683ded7b"
server_key_iv: "a4f954a1163dd39ed6bcd231a3d3b861"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.359357 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:47.378073 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:47.378973 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:47.379482 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:47.380255 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:47.381345 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:47.381394 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.381462 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:47.381494 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.392803 16755 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.129:43729
I20260924 20:18:47.392820 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.129:43729 every 8 connection(s)
I20260924 20:18:47.393880 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/data/info.pb
I20260924 20:18:47.398455 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16755
I20260924 20:18:47.398603 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-0/wal/instance
I20260924 20:18:47.398959 30030 external_mini_cluster.cc:1428] Setting key f0f01c6a11ecb8ba7e69595e4217c751
I20260924 20:18:47.401180 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.130:0
--local_ip_for_outbound_sockets=127.29.83.130
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--builtin_ntp_servers=127.29.83.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:47.405552 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.396072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.129:36617 (local address 127.29.83.190:45509)
0924 20:18:47.396205 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.396216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.397730 (+ 1514us) server_negotiation.cc:407] Connection header received
0924 20:18:47.399465 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.399469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.399529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.399607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:47.400956 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.401989 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.403432 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.403661 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.403800 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.405023 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:47.405027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:47.405030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:47.405052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:47.405072 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:47.405079 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:47.405158 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.405336 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.405406 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:18:47.406651 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.396724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45509 (local address 127.29.83.129:36617)
0924 20:18:47.397540 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.397585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.399222 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.399923 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.399933 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.400048 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:47.400778 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.400801 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.402143 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.402150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.403291 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.403301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.403861 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.403866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.404050 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.404730 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:47.404731 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:47.404776 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:47.404845 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:47.405160 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:47.405172 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:47.405180 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.405325 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.405888 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":144,"threads_started":1}
I20260924 20:18:47.408300 16883 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45509
I20260924 20:18:47.408569 16883 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:47.409241 16883 heartbeater.cc:511] Master 127.29.83.190:45509 requested a full tablet report, sending...
I20260924 20:18:47.410817 16695 ts_manager.cc:194] Registered new tserver with Master: 09a0f0de1546450199f75028a0da0ded (127.29.83.129:43729)
I20260924 20:18:47.411998 16695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.129:36617
W20260924 20:18:47.526561 16886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:47.526870 16886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:47.526909 16886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:47.531061 16886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:47.531131 16886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:47.531190 16886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.130
I20260924 20:18:47.536021 16886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.83.130
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.130
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:47.537093 16886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:47.538635 16886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:47.548652 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.548784 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.548652 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:47.548909 16886 server_base.cc:1034] running on GCE node
I20260924 20:18:47.549417 16886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:47.550190 16886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:47.551388 16886 hybrid_clock.cc:648] HybridClock initialized: now 1790281127551347 us; error 54 us; skew 500 ppm
I20260924 20:18:47.553498 16886 webserver.cc:460] Webserver started at http://127.29.83.130:45043/ using document root <none> and password file <none>
I20260924 20:18:47.554177 16886 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:47.554244 16886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:47.554420 16886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:47.556519 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data/instance:
uuid: "eb49583babd34de7b3f38ceb19eea353"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "1aea51d95b088a0871cba5d27ca4f3e3"
server_key_iv: "4222de5995044bd988924de9c417031b"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.557207 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance:
uuid: "eb49583babd34de7b3f38ceb19eea353"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "1aea51d95b088a0871cba5d27ca4f3e3"
server_key_iv: "4222de5995044bd988924de9c417031b"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.561050 16886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:18:47.564000 16900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.565342 16886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 20:18:47.565478 16886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal
uuid: "eb49583babd34de7b3f38ceb19eea353"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "1aea51d95b088a0871cba5d27ca4f3e3"
server_key_iv: "4222de5995044bd988924de9c417031b"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.565582 16886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:47.578240 16886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:47.578958 16886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:47.579113 16886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:47.579650 16886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:47.580783 16886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:47.580847 16886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.580893 16886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:47.580919 16886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.590938 16886 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.130:46179
I20260924 20:18:47.591029 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.130:46179 every 8 connection(s)
I20260924 20:18:47.591938 16886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/data/info.pb
I20260924 20:18:47.599483 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 16886
I20260924 20:18:47.599668 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-1/wal/instance
I20260924 20:18:47.599975 30030 external_mini_cluster.cc:1428] Setting key 30c07bf37122a0225be18ff8568ed9c9
I20260924 20:18:47.602317 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.83.131:0
--local_ip_for_outbound_sockets=127.29.83.131
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--builtin_ntp_servers=127.29.83.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:18:47.602401 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.593989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:48201 (local address 127.29.83.190:45509)
0924 20:18:47.594151 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.594154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.595309 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 20:18:47.596948 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.596952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.597016 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.597129 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:47.598182 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.598730 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.600064 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.600286 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.600402 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.601752 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:47.601789 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:47.601792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:47.601808 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:47.601832 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:47.601844 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:47.601938 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.602140 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.602236 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:18:47.603440 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.594374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45509 (local address 127.29.83.130:48201)
0924 20:18:47.595112 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.595157 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.596706 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.597327 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.597334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.597432 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:47.598029 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.598045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.598867 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.598873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.599929 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.599940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.600452 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.600456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.600683 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.601416 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:47.601419 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:47.601478 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:47.601561 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:47.601968 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:47.601977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:47.601983 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.602110 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.602369 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":106,"threads_started":1}
I20260924 20:18:47.605109 17014 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45509
I20260924 20:18:47.605372 17014 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:47.605990 17014 heartbeater.cc:511] Master 127.29.83.190:45509 requested a full tablet report, sending...
I20260924 20:18:47.607052 16695 ts_manager.cc:194] Registered new tserver with Master: eb49583babd34de7b3f38ceb19eea353 (127.29.83.130:46179)
I20260924 20:18:47.607712 16695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.130:48201
W20260924 20:18:47.718875 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:47.719172 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:47.719246 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:47.723261 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:47.723349 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:47.723439 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.83.131
I20260924 20:18:47.728206 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.83.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.83.131
--webserver_port=0
--tserver_master_addrs=127.29.83.190:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.83.131
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:47.729311 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:47.730733 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:47.740211 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.740211 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:47.740250 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:47.740365 17017 server_base.cc:1034] running on GCE node
I20260924 20:18:47.740949 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:47.741602 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:47.742769 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790281127742748 us; error 30 us; skew 500 ppm
I20260924 20:18:47.744719 17017 webserver.cc:460] Webserver started at http://127.29.83.131:43453/ using document root <none> and password file <none>
I20260924 20:18:47.745302 17017 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:47.745363 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:47.745599 17017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:47.747679 17017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data/instance:
uuid: "83fa16de83ed4ad7b1121eea85c57eb9"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "d65f813cfa06265af1ca1d3db5e7490d"
server_key_iv: "a2fb74aa4938ce9fde4ae095043b5b53"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.748255 17017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance:
uuid: "83fa16de83ed4ad7b1121eea85c57eb9"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "d65f813cfa06265af1ca1d3db5e7490d"
server_key_iv: "a2fb74aa4938ce9fde4ae095043b5b53"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.751964 17017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:18:47.754595 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.755882 17017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:18:47.756006 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal
uuid: "83fa16de83ed4ad7b1121eea85c57eb9"
format_stamp: "Formatted at 2026-09-24 20:18:47 on dist-test-slave-52fr"
server_key: "d65f813cfa06265af1ca1d3db5e7490d"
server_key_iv: "a2fb74aa4938ce9fde4ae095043b5b53"
server_key_version: "encryptionkey@0"
I20260924 20:18:47.756115 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:47.778643 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:47.779400 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:47.779628 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:47.780205 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:18:47.781111 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:18:47.781158 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.781198 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:18:47.781213 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:47.790570 17017 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.131:38545
I20260924 20:18:47.790589 17144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.131:38545 every 8 connection(s)
I20260924 20:18:47.791498 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/data/info.pb
I20260924 20:18:47.800134 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17017
I20260924 20:18:47.800243 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal/instance
I20260924 20:18:47.800596 30030 external_mini_cluster.cc:1428] Setting key fc75ab16d02c0c70dbe037179fcd6327
I20260924 20:18:47.801371 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.793380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.131:55569 (local address 127.29.83.190:45509)
0924 20:18:47.793534 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.793539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.794782 (+ 1243us) server_negotiation.cc:407] Connection header received
0924 20:18:47.796167 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.796170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.796219 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.796355 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:47.797484 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.798074 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.799286 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.799511 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.799592 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.800860 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:47.800863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:47.800865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:47.800877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:47.800914 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:47.800927 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:47.801007 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.801166 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.801229 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:18:47.802389 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.793921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45509 (local address 127.29.83.131:55569)
0924 20:18:47.794600 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.794642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.795956 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.796683 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.796691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.796780 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:18:47.797289 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.797304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.798195 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.798198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.799173 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.799183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.799645 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.799647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.799782 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.800558 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:18:47.800560 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:18:47.800614 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:18:47.800698 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:18:47.801017 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:18:47.801024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:18:47.801029 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.801161 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.801482 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
I20260924 20:18:47.803434 17145 heartbeater.cc:347] Connected to a master server at 127.29.83.190:45509
I20260924 20:18:47.803682 17145 heartbeater.cc:464] Registering TS with master...
I20260924 20:18:47.804296 17145 heartbeater.cc:511] Master 127.29.83.190:45509 requested a full tablet report, sending...
I20260924 20:18:47.805500 16695 ts_manager.cc:194] Registered new tserver with Master: 83fa16de83ed4ad7b1121eea85c57eb9 (127.29.83.131:38545)
I20260924 20:18:47.806223 16695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.83.131:55569
I20260924 20:18:47.814627 30030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:18:47.820565 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.816612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.29.83.190:45509)
0924 20:18:47.816804 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.816809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.816940 (+ 131us) server_negotiation.cc:407] Connection header received
0924 20:18:47.817094 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.817098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.817146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.817234 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:47.817505 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.818101 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.818961 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.819175 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.819286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.820038 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:47.820043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:47.820046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:47.820062 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:47.820080 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:47.820087 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:47.820169 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.820359 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.820416 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:47.823740 16695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:18:47.826768 16695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:18:47.842135 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.836111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.29.83.129:43729)
0924 20:18:47.837117 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.837132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.837162 (+ 30us) server_negotiation.cc:407] Connection header received
0924 20:18:47.837262 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.837268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.837492 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.837691 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:47.838050 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839185 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.840757 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.841381 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.841515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.841734 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.841867 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.841969 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":120,"threads_started":1}
I20260924 20:18:47.842131 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.835851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:43729 (local address 127.0.0.1:42432)
0924 20:18:47.836839 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.836896 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.837046 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.837682 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.837690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.837713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:47.837913 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.837923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839323 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.839329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.840629 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.840639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.841512 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.841516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.841633 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.841670 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.841836 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.841969 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260924 20:18:47.842563 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.836665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:38545 (local address 127.0.0.1:46012)
0924 20:18:47.837763 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.837795 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.837872 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.838442 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.838445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.838463 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:47.838541 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.838546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839849 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.839852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.840830 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.840841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.842126 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.842130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.842219 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.842242 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.842321 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.842398 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260924 20:18:47.842949 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.835854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:46179 (local address 127.0.0.1:55156)
0924 20:18:47.837479 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.837489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.837569 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.838130 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.838134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.838153 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:47.838237 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.838244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839506 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.839510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.841444 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.841454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.842554 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.842560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.842656 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.842680 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.842762 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.842827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":548,"thread_start_us":93,"threads_started":1}
I20260924 20:18:47.843117 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.835938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.29.83.130:46179)
0924 20:18:47.836367 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.836382 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.837591 (+ 1209us) server_negotiation.cc:407] Connection header received
0924 20:18:47.837688 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.837694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.837945 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.838135 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:47.838337 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839357 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.841587 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.842393 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.842523 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.842713 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.842870 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.842966 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":158,"threads_started":1}
I20260924 20:18:47.843176 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.836992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.29.83.131:38545)
0924 20:18:47.837375 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.837389 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.837887 (+ 498us) server_negotiation.cc:407] Connection header received
0924 20:18:47.837968 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.837972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.838144 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.838294 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:47.838670 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.839719 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.840961 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.841790 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.841887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.842065 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.842934 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.843020 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":157,"threads_started":1}
I20260924 20:18:47.845638 16948 tablet_service.cc:1467] Processing CreateTablet for tablet 81b5997216934ef8b33bb2ded8f8696f (DEFAULT_TABLE table=test-table [id=f5ae8d4ccaa944dda559fae8a733190b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:47.845600 17079 tablet_service.cc:1467] Processing CreateTablet for tablet 81b5997216934ef8b33bb2ded8f8696f (DEFAULT_TABLE table=test-table [id=f5ae8d4ccaa944dda559fae8a733190b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:47.845638 16817 tablet_service.cc:1467] Processing CreateTablet for tablet 81b5997216934ef8b33bb2ded8f8696f (DEFAULT_TABLE table=test-table [id=f5ae8d4ccaa944dda559fae8a733190b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:18:47.846779 17079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b5997216934ef8b33bb2ded8f8696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:47.846781 16948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b5997216934ef8b33bb2ded8f8696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:47.846781 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b5997216934ef8b33bb2ded8f8696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:47.853492 17162 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Bootstrap starting.
I20260924 20:18:47.854480 17164 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Bootstrap starting.
I20260924 20:18:47.857193 17164 tablet_bootstrap.cc:654] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:47.857299 17162 tablet_bootstrap.cc:654] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:47.858237 17163 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Bootstrap starting.
I20260924 20:18:47.858256 17162 log.cc:824] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:47.858256 17164 log.cc:824] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:47.860536 17164 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: No bootstrap required, opened a new log
I20260924 20:18:47.860536 17162 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: No bootstrap required, opened a new log
I20260924 20:18:47.860751 17164 ts_tablet_manager.cc:1397] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:18:47.860751 17162 ts_tablet_manager.cc:1397] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:18:47.861006 17163 tablet_bootstrap.cc:654] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:47.861990 17163 log.cc:824] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:47.864256 17163 tablet_bootstrap.cc:492] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: No bootstrap required, opened a new log
I20260924 20:18:47.864475 17163 ts_tablet_manager.cc:1397] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:18:47.865350 17162 raft_consensus.cc:348] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.865342 17164 raft_consensus.cc:348] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.865635 17162 raft_consensus.cc:374] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:47.865634 17164 raft_consensus.cc:374] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:47.865692 17164 raft_consensus.cc:729] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0f0de1546450199f75028a0da0ded, State: Initialized, Role: FOLLOWER
I20260924 20:18:47.865692 17162 raft_consensus.cc:729] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fa16de83ed4ad7b1121eea85c57eb9, State: Initialized, Role: FOLLOWER
I20260924 20:18:47.866201 17162 consensus_queue.cc:260] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.866201 17164 consensus_queue.cc:260] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.866969 17145 heartbeater.cc:503] Master 127.29.83.190:45509 was elected leader, sending a full tablet report...
I20260924 20:18:47.867266 17162 ts_tablet_manager.cc:1428] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260924 20:18:47.867252 16883 heartbeater.cc:503] Master 127.29.83.190:45509 was elected leader, sending a full tablet report...
I20260924 20:18:47.867266 17164 ts_tablet_manager.cc:1428] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:18:47.870304 17163 raft_consensus.cc:348] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.870616 17163 raft_consensus.cc:374] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:47.870682 17163 raft_consensus.cc:729] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb49583babd34de7b3f38ceb19eea353, State: Initialized, Role: FOLLOWER
I20260924 20:18:47.871168 17163 consensus_queue.cc:260] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.871876 17014 heartbeater.cc:503] Master 127.29.83.190:45509 was elected leader, sending a full tablet report...
I20260924 20:18:47.872344 17163 ts_tablet_manager.cc:1428] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260924 20:18:47.895505 16884 tablet.cc:2369] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:18:47.897910 17170 raft_consensus.cc:482] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:18:47.898146 17170 raft_consensus.cc:504] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.899436 17170 leader_election.cc:288] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a0f0de1546450199f75028a0da0ded (127.29.83.129:43729), 83fa16de83ed4ad7b1121eea85c57eb9 (127.29.83.131:38545)
I20260924 20:18:47.904399 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.899850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:59601 (local address 127.29.83.131:38545)
0924 20:18:47.899979 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.899983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.899997 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:47.900148 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.900155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.900238 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.900314 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:47.900608 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.901356 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.902959 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.903667 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.903794 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.903870 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.904170 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.904255 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:18:47.904393 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.899784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.83.130:41041 (local address 127.29.83.129:43729)
0924 20:18:47.899925 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.899930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.900205 (+ 275us) server_negotiation.cc:407] Connection header received
0924 20:18:47.900441 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.900444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.900500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.900552 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:18:47.900982 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.901628 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.903162 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.903784 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.903896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.903942 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.904170 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.904256 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:18:47.904441 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.899689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.131:38545 (local address 127.29.83.130:59601)
0924 20:18:47.899898 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.899916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.900017 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.900337 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.900343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.900362 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:47.900497 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.900504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.901489 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.901492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.902810 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.902821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.903809 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.903813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.903955 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.903987 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.904089 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.904248 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 20:18:47.904650 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.899695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:43729 (local address 127.29.83.130:41041)
0924 20:18:47.900103 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.900114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:18:47.900302 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:18:47.900686 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:18:47.900688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:18:47.900702 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:18:47.900841 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.900846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.901780 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.901784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.902949 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:18:47.902956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.903905 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.903909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:18:47.903996 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.904022 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:18:47.904102 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:18:47.904248 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":131,"threads_started":1}
I20260924 20:18:47.905169 17099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b5997216934ef8b33bb2ded8f8696f" candidate_uuid: "eb49583babd34de7b3f38ceb19eea353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" is_pre_election: true
I20260924 20:18:47.905274 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b5997216934ef8b33bb2ded8f8696f" candidate_uuid: "eb49583babd34de7b3f38ceb19eea353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0f0de1546450199f75028a0da0ded" is_pre_election: true
I20260924 20:18:47.905524 17099 raft_consensus.cc:2454] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49583babd34de7b3f38ceb19eea353 in term 0.
I20260924 20:18:47.905524 16837 raft_consensus.cc:2454] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49583babd34de7b3f38ceb19eea353 in term 0.
I20260924 20:18:47.906072 16903 leader_election.cc:302] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fa16de83ed4ad7b1121eea85c57eb9, eb49583babd34de7b3f38ceb19eea353; no voters:
I20260924 20:18:47.906322 17170 raft_consensus.cc:2781] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:18:47.906414 17170 raft_consensus.cc:482] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:18:47.906461 17170 raft_consensus.cc:3037] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:47.907682 17170 raft_consensus.cc:504] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.908109 17170 leader_election.cc:288] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [CANDIDATE]: Term 1 election: Requested vote from peers 09a0f0de1546450199f75028a0da0ded (127.29.83.129:43729), 83fa16de83ed4ad7b1121eea85c57eb9 (127.29.83.131:38545)
I20260924 20:18:47.908445 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b5997216934ef8b33bb2ded8f8696f" candidate_uuid: "eb49583babd34de7b3f38ceb19eea353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0f0de1546450199f75028a0da0ded"
I20260924 20:18:47.908597 17099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b5997216934ef8b33bb2ded8f8696f" candidate_uuid: "eb49583babd34de7b3f38ceb19eea353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fa16de83ed4ad7b1121eea85c57eb9"
I20260924 20:18:47.908607 16837 raft_consensus.cc:3037] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:47.908773 17099 raft_consensus.cc:3037] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:47.909880 16837 raft_consensus.cc:2454] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49583babd34de7b3f38ceb19eea353 in term 1.
I20260924 20:18:47.910111 17099 raft_consensus.cc:2454] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49583babd34de7b3f38ceb19eea353 in term 1.
I20260924 20:18:47.910338 16901 leader_election.cc:302] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0f0de1546450199f75028a0da0ded, eb49583babd34de7b3f38ceb19eea353; no voters:
I20260924 20:18:47.910513 17170 raft_consensus.cc:2781] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:47.910715 17170 raft_consensus.cc:686] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 1 LEADER]: Becoming Leader. State: Replica: eb49583babd34de7b3f38ceb19eea353, State: Running, Role: LEADER
I20260924 20:18:47.911052 17170 consensus_queue.cc:237] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } }
I20260924 20:18:47.914472 16694 catalog_manager.cc:5495] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb49583babd34de7b3f38ceb19eea353 (127.29.83.130). New cstate: current_term: 1 leader_uuid: "eb49583babd34de7b3f38ceb19eea353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb49583babd34de7b3f38ceb19eea353" member_type: VOTER last_known_addr { host: "127.29.83.130" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 } health_report { overall_health: UNKNOWN } } }
I20260924 20:18:47.945447 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.941704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.29.83.130:46179)
0924 20:18:47.941905 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:47.941908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:47.941922 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:18:47.942002 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:47.942004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:47.942058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:47.942154 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:18:47.942373 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.942864 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.943843 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:47.944054 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:47.944142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:47.944473 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:18:47.945010 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:18:47.945113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:47.945193 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 20:18:47.945266 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 20:18:47.951880 16837 raft_consensus.cc:1261] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded [term 1 FOLLOWER]: Refusing update from remote peer eb49583babd34de7b3f38ceb19eea353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:47.951891 17099 raft_consensus.cc:1261] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 1 FOLLOWER]: Refusing update from remote peer eb49583babd34de7b3f38ceb19eea353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:18:47.952726 17170 consensus_queue.cc:1035] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0f0de1546450199f75028a0da0ded" member_type: VOTER last_known_addr { host: "127.29.83.129" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:47.952992 17172 consensus_queue.cc:1035] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fa16de83ed4ad7b1121eea85c57eb9" member_type: VOTER last_known_addr { host: "127.29.83.131" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:18:47.961665 17177 mvcc.cc:205] Tried to move back new op lower bound from 7332991500082499584 to 7332991499926409216. Current Snapshot: MvccSnapshot[applied={T|T < 7332991500082499584}]
I20260924 20:18:47.962699 17176 mvcc.cc:205] Tried to move back new op lower bound from 7332991500082499584 to 7332991499926409216. Current Snapshot: MvccSnapshot[applied={T|T < 7332991500082499584}]
I20260924 20:18:47.971052 16694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:47.971259 16694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:18:47.974004 16694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda048b16c944923a58eab13b9e39372: Sending DeleteTablet for 3 replicas of tablet 81b5997216934ef8b33bb2ded8f8696f
I20260924 20:18:47.974959 17079 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b5997216934ef8b33bb2ded8f8696f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:47 UTC) from {username='slave'} at 127.0.0.1:46012
I20260924 20:18:47.974999 16948 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b5997216934ef8b33bb2ded8f8696f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:47 UTC) from {username='slave'} at 127.0.0.1:55156
I20260924 20:18:47.975555 17185 tablet_replica.cc:330] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: stopping tablet replica
I20260924 20:18:47.975571 16817 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b5997216934ef8b33bb2ded8f8696f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:18:47 UTC) from {username='slave'} at 127.0.0.1:42432
I20260924 20:18:47.976017 17187 tablet_replica.cc:330] T 81b5997216934ef8b33bb2ded8f8696f P 09a0f0de1546450199f75028a0da0ded: stopping tablet replica
I20260924 20:18:47.976061 17185 raft_consensus.cc:2229] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:18:47.976413 17186 tablet_replica.cc:330] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: stopping tablet replica
I20260924 20:18:47.976567 17185 raft_consensus.cc:2258] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:47.976768 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16755
I20260924 20:18:47.978302 17186 raft_consensus.cc:2229] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:18:47.978715 17186 raft_consensus.cc:2258] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:18:47.980062 17185 ts_tablet_manager.cc:1905] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:18:47.983218 16679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.83.129:43729 (error 108)
I20260924 20:18:47.984117 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16886
I20260924 20:18:47.984476 17186 ts_tablet_manager.cc:1905] T 81b5997216934ef8b33bb2ded8f8696f P eb49583babd34de7b3f38ceb19eea353: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:18:47.984840 17185 ts_tablet_manager.cc:1918] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:18:47.984936 17185 log.cc:1196] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Deleting WAL directory at /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280784619584-30030-0/minicluster-data/ts-2/wal/wals/81b5997216934ef8b33bb2ded8f8696f
I20260924 20:18:47.985248 17185 ts_tablet_manager.cc:1939] T 81b5997216934ef8b33bb2ded8f8696f P 83fa16de83ed4ad7b1121eea85c57eb9: Deleting consensus metadata
I20260924 20:18:47.986332 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.986027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.129:43729 (local address 127.0.0.1:42436)
0924 20:18:47.986168 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.986252 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.129:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 20:18:47.986555 16679 catalog_manager.cc:4584] TS 09a0f0de1546450199f75028a0da0ded (127.29.83.129:43729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81b5997216934ef8b33bb2ded8f8696f: Network error: Client connection negotiation failed: client connection to 127.29.83.129:43729: connect: Connection refused (error 111)
I20260924 20:18:47.986610 16681 catalog_manager.cc:4841] TS 83fa16de83ed4ad7b1121eea85c57eb9 (127.29.83.131:38545): tablet 81b5997216934ef8b33bb2ded8f8696f (table test-table [id=f5ae8d4ccaa944dda559fae8a733190b]) successfully deleted
I20260924 20:18:47.990465 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17017
I20260924 20:18:47.991704 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:47.990622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.130:46179 (local address 127.0.0.1:55188)
0924 20:18:47.991527 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 20:18:47.991598 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.130:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":800}
I20260924 20:18:47.996168 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 16664
2026-09-24T20:18:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1119 ms)
[----------] 6 tests from EncryptionPolicyTest (11107 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:48 dist-test-slave-52fr krb5kdc[17191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:48 dist-test-slave-52fr krb5kdc[17191](info): set up 2 sockets
Sep 24 20:18:48 dist-test-slave-52fr krb5kdc[17191](info): commencing operation
krb5kdc: starting...
W20260924 20:18:49.042124 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:49.173100 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:49.285178 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:49.285524 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:49.285609 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:49.289691 17204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:49.289820 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:49.289867 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:49.289907 17204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:49.289942 17204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:49.295497 17204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:49.296693 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:49.297804 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:49.306571 17212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:49.306792 17204 server_base.cc:1034] running on GCE node
W20260924 20:18:49.306583 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:49.306593 17209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:49.307547 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:49.308672 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:49.309873 17204 hybrid_clock.cc:648] HybridClock initialized: now 1790281129309849 us; error 35 us; skew 500 ppm
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:49.312839 17204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:49.314004 17204 webserver.cc:460] Webserver started at http://127.0.0.1:35543/ using document root <none> and password file <none>
I20260924 20:18:49.314589 17204 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:49.314684 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:49.314970 17204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:49.317018 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "1cf5cf87d4cc49f0b45db455a201f9b2"
format_stamp: "Formatted at 2026-09-24 20:18:49 on dist-test-slave-52fr"
server_key: "b0b10567740aea062be7921a6467a853"
server_key_iv: "814b6179fabb150020ba8ea78c97eaaf"
server_key_version: "encryptionkey@0"
I20260924 20:18:49.317605 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "1cf5cf87d4cc49f0b45db455a201f9b2"
format_stamp: "Formatted at 2026-09-24 20:18:49 on dist-test-slave-52fr"
server_key: "b0b10567740aea062be7921a6467a853"
server_key_iv: "814b6179fabb150020ba8ea78c97eaaf"
server_key_version: "encryptionkey@0"
I20260924 20:18:49.321405 17204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:18:49.324455 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:49.326006 17204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:49.326161 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "1cf5cf87d4cc49f0b45db455a201f9b2"
format_stamp: "Formatted at 2026-09-24 20:18:49 on dist-test-slave-52fr"
server_key: "b0b10567740aea062be7921a6467a853"
server_key_iv: "814b6179fabb150020ba8ea78c97eaaf"
server_key_version: "encryptionkey@0"
I20260924 20:18:49.326298 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:49.349448 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:49.352741 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:49.352984 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:49.360546 17204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46561
I20260924 20:18:49.360556 17271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46561 every 8 connection(s)
I20260924 20:18:49.361572 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:49.364544 17272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:49.370078 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17204
I20260924 20:18:49.370231 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:49.370306 17272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: Bootstrap starting.
I20260924 20:18:49.370540 30030 external_mini_cluster.cc:1428] Setting key 9a9b2f4d5e20c02c01cdb8304e4d8279
I20260924 20:18:49.372898 17272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:49.373697 17272 log.cc:824] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:49.375674 17272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: No bootstrap required, opened a new log
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:18:49.380081 17272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260924 20:18:49.380332 17272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:49.380412 17272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf5cf87d4cc49f0b45db455a201f9b2, State: Initialized, Role: FOLLOWER
I20260924 20:18:49.380929 17272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260924 20:18:49.381091 17272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:49.381170 17272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:49.381284 17272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:49.382247 17272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260924 20:18:49.382596 17272 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf5cf87d4cc49f0b45db455a201f9b2; no voters:
I20260924 20:18:49.382931 17272 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:49.383030 17277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:49.383216 17277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf5cf87d4cc49f0b45db455a201f9b2, State: Running, Role: LEADER
I20260924 20:18:49.383513 17277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } }
I20260924 20:18:49.384260 17272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:49.385859 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:49.371875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37603 (local address 127.0.0.1:46561)
0924 20:18:49.372268 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:49.372293 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:49.372329 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:18:49.372991 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:49.373017 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:49.373367 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:49.373645 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:49.374567 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:49.375547 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:49.376681 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:49.377136 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:49.377262 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:49.380185 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:49.380190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:49.380203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:49.380230 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:49.382154 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:49.382698 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:49.382703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:49.382708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:49.382855 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:49.383238 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:49.383242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:49.383244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:49.383757 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:49.383945 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:49.384590 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 20:18:49.384835 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260924 20:18:49.386624 17278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260924 20:18:49.386756 17278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:49.387266 17279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf5cf87d4cc49f0b45db455a201f9b2. Latest consensus state: current_term: 1 leader_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5cf87d4cc49f0b45db455a201f9b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46561 } } }
I20260924 20:18:49.387362 17279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:49.389180 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 20:18:49.389969 17292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:18:49.390069 17292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:18:49.390180 17293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:49.391075 17293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:49.397051 17293 catalog_manager.cc:1347] Generated new cluster ID: dd75aa706978437880a480189de6bf3e
I20260924 20:18:49.397113 17293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:49.403326 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:49.394174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50131 (local address 127.0.0.1:46561)
0924 20:18:49.394321 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:49.394325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:49.394537 (+ 212us) server_negotiation.cc:407] Connection header received
0924 20:18:49.394644 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:49.394647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:49.394691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:49.394779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:49.395716 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:49.396488 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:49.397667 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:49.398003 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:49.398101 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:49.399751 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:49.399754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:49.399757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:49.399780 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:49.401457 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:49.402036 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:49.402040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:49.402042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:49.402121 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:49.402455 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:49.402458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:49.402460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:49.402647 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:49.402775 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:49.403068 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:18:49.403160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 20:18:49.419688 17293 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:49.420715 17293 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:49.430495 17293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf5cf87d4cc49f0b45db455a201f9b2: Generated new TSK 0
I20260924 20:18:49.431370 17293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:49.435791 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17204
W20260924 20:18:49.441461 17189 connection.cc:535] client connection to 127.0.0.1:46561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46561): Connection reset by peer (error 104)
2026-09-24T20:18:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17303](info): set up 2 sockets
Sep 24 20:18:49 dist-test-slave-52fr krb5kdc[17303](info): commencing operation
krb5kdc: starting...
W20260924 20:18:50.482059 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:50 dist-test-slave-52fr krb5kdc[17303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:50.610085 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:50.725623 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:50.725989 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:50.726094 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:50.730322 17316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:50.730453 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:50.730499 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:50.730544 17316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:50.730585 17316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:50.736271 17316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:50.737461 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:50.738602 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:50.747485 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:50.747485 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:50.747504 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:50.748111 17316 server_base.cc:1034] running on GCE node
I20260924 20:18:50.748636 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:50.749740 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:50.751021 17316 hybrid_clock.cc:648] HybridClock initialized: now 1790281130750992 us; error 31 us; skew 500 ppm
Sep 24 20:18:50 dist-test-slave-52fr krb5kdc[17303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:50.754001 17316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:50.754881 17316 webserver.cc:460] Webserver started at http://127.0.0.1:39979/ using document root <none> and password file <none>
I20260924 20:18:50.755461 17316 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:50.755559 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:50.755795 17316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:50.757856 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "726c3135b831401888646d6cb81fef15"
format_stamp: "Formatted at 2026-09-24 20:18:50 on dist-test-slave-52fr"
server_key: "910c4bfd92cf9056dd6ed38a4b65f45b"
server_key_iv: "3ed84d474c458b768ffff60c137e1906"
server_key_version: "encryptionkey@0"
I20260924 20:18:50.758432 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "726c3135b831401888646d6cb81fef15"
format_stamp: "Formatted at 2026-09-24 20:18:50 on dist-test-slave-52fr"
server_key: "910c4bfd92cf9056dd6ed38a4b65f45b"
server_key_iv: "3ed84d474c458b768ffff60c137e1906"
server_key_version: "encryptionkey@0"
I20260924 20:18:50.762215 17316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 20:18:50.764868 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:50.766517 17316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:50.766676 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "726c3135b831401888646d6cb81fef15"
format_stamp: "Formatted at 2026-09-24 20:18:50 on dist-test-slave-52fr"
server_key: "910c4bfd92cf9056dd6ed38a4b65f45b"
server_key_iv: "3ed84d474c458b768ffff60c137e1906"
server_key_version: "encryptionkey@0"
I20260924 20:18:50.766803 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:50.796104 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:50.804878 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:50.805114 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:50.812855 17383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36919 every 8 connection(s)
I20260924 20:18:50.812855 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36919
I20260924 20:18:50.814185 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:50.817090 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17316
I20260924 20:18:50.817261 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:50.817303 17384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:50.817716 30030 external_mini_cluster.cc:1428] Setting key bb2661d7b8e5ba7cf744f9a0614fde71
I20260924 20:18:50.825301 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15: Bootstrap starting.
I20260924 20:18:50.827992 17384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:50.828395 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:50.819396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37541 (local address 127.0.0.1:36919)
0924 20:18:50.819841 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:50.819859 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:50.819902 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:18:50.820784 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:50.820811 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:50.821261 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:50.821646 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:50.822102 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:50.823632 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:50.824505 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:50.825181 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:50.825347 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:50.826339 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:50.826345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:50.826362 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:50.826891 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:50.826953 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:50.826983 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:50.827106 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:50.827308 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 20:18:50.827440 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260924 20:18:50.828840 17384 log.cc:824] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:50.831183 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15: No bootstrap required, opened a new log
I20260924 20:18:50.831501 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:50.835350 17384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } }
I20260924 20:18:50.835629 17384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:50.835676 17384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726c3135b831401888646d6cb81fef15, State: Initialized, Role: FOLLOWER
I20260924 20:18:50.836138 17384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } }
I20260924 20:18:50.836264 17384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:50.836350 17384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:50.836457 17384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:50.837534 17384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } }
I20260924 20:18:50.837811 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:50.833586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51591 (local address 127.0.0.1:36919)
0924 20:18:50.833705 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:50.833709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:50.833861 (+ 152us) server_negotiation.cc:407] Connection header received
0924 20:18:50.833983 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:50.833986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:50.834022 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:50.834081 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:50.834408 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:50.834918 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:50.836025 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:50.836272 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:50.836407 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:50.837268 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:50.837271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:50.837273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:50.837284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:50.837306 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:50.837316 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:50.837409 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:50.837592 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 20:18:50.837629 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:18:50.837954 17384 leader_election.cc:302] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 726c3135b831401888646d6cb81fef15; no voters:
I20260924 20:18:50.838335 17384 leader_election.cc:288] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:50.838433 17394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:50.838642 17394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [term 1 LEADER]: Becoming Leader. State: Replica: 726c3135b831401888646d6cb81fef15, State: Running, Role: LEADER
I20260924 20:18:50.838976 17394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } }
I20260924 20:18:50.839576 17384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:50.840094 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 726c3135b831401888646d6cb81fef15. Latest consensus state: current_term: 1 leader_uuid: "726c3135b831401888646d6cb81fef15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } } }
I20260924 20:18:50.840273 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:50.840605 17395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "726c3135b831401888646d6cb81fef15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726c3135b831401888646d6cb81fef15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36919 } } }
I20260924 20:18:50.840791 17395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:50.840692 17401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:50.844432 17401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:50.850003 17401 catalog_manager.cc:1347] Generated new cluster ID: 81bbbdfb79de472fab9bb987c825ab06
I20260924 20:18:50.850088 17401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:50.861956 17401 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:50.863260 17401 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:50.871280 17401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 726c3135b831401888646d6cb81fef15: Generated new TSK 0
I20260924 20:18:50.872040 17401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:50.877820 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17316
W20260924 20:18:50.883224 17301 connection.cc:535] client connection to 127.0.0.1:36919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36919): Connection reset by peer (error 104)
2026-09-24T20:18:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:50 dist-test-slave-52fr krb5kdc[17415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:50 dist-test-slave-52fr krb5kdc[17415](info): set up 2 sockets
Sep 24 20:18:50 dist-test-slave-52fr krb5kdc[17415](info): commencing operation
krb5kdc: starting...
W20260924 20:18:51.918159 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:52 dist-test-slave-52fr krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:52.045187 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:52.155195 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:52.155491 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:52.155572 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:52.159473 17428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:52.159569 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:52.159606 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:52.159641 17428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:52.159677 17428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:52.164815 17428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:52.165942 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:52.166976 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:52.175657 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:52.175645 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:52.175645 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:52.175714 17428 server_base.cc:1034] running on GCE node
I20260924 20:18:52.176457 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:52.177578 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:52.178742 17428 hybrid_clock.cc:648] HybridClock initialized: now 1790281132178720 us; error 40 us; skew 500 ppm
Sep 24 20:18:52 dist-test-slave-52fr krb5kdc[17415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:52.181524 17428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:52.182523 17428 webserver.cc:460] Webserver started at http://127.0.0.1:44141/ using document root <none> and password file <none>
I20260924 20:18:52.183110 17428 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:52.183204 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:52.183440 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:52.185497 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "d7298a1ad7c34c70b5ad976e47bb846f"
format_stamp: "Formatted at 2026-09-24 20:18:52 on dist-test-slave-52fr"
server_key: "2cd31913c06efa07668a54fe05c78942"
server_key_iv: "84a2c0861f7fd37586d425f023da5264"
server_key_version: "encryptionkey@0"
I20260924 20:18:52.186110 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "d7298a1ad7c34c70b5ad976e47bb846f"
format_stamp: "Formatted at 2026-09-24 20:18:52 on dist-test-slave-52fr"
server_key: "2cd31913c06efa07668a54fe05c78942"
server_key_iv: "84a2c0861f7fd37586d425f023da5264"
server_key_version: "encryptionkey@0"
I20260924 20:18:52.189857 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:18:52.192366 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:52.193650 17428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:52.193822 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "d7298a1ad7c34c70b5ad976e47bb846f"
format_stamp: "Formatted at 2026-09-24 20:18:52 on dist-test-slave-52fr"
server_key: "2cd31913c06efa07668a54fe05c78942"
server_key_iv: "84a2c0861f7fd37586d425f023da5264"
server_key_version: "encryptionkey@0"
I20260924 20:18:52.193958 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:52.198698 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:52.201808 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:52.202005 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:52.208947 17428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39203
I20260924 20:18:52.209056 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39203 every 8 connection(s)
I20260924 20:18:52.210049 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:52.211272 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17428
I20260924 20:18:52.211436 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:52.211753 30030 external_mini_cluster.cc:1428] Setting key 06f93339ea44d02d4ca07ed42feda368
I20260924 20:18:52.214283 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:52.217303 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:52.213294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33059 (local address 127.0.0.1:39203)
0924 20:18:52.213695 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:52.213714 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:52.213809 (+ 95us) server_negotiation.cc:407] Connection header received
0924 20:18:52.214523 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:52.214542 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:52.214831 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:52.215109 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:52.216009 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:52.216014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:52.216026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:52.216038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:52.216095 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:52.216120 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:52.216276 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:52.216368 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 20:18:52.216554 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20260924 20:18:52.220543 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:52.221025 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f: Bootstrap starting.
I20260924 20:18:52.223953 17496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:52.224782 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:52.222981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32935 (local address 127.0.0.1:39203)
0924 20:18:52.223115 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:52.223120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:52.223301 (+ 181us) server_negotiation.cc:407] Connection header received
0924 20:18:52.223389 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:52.223392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:52.223429 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:52.223514 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:52.224353 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:52.224356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:52.224358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:52.224368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:52.224395 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:52.224405 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:52.224475 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:52.224591 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 20:18:52.224623 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:18:52.224916 17496 log.cc:824] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:52.227092 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f: No bootstrap required, opened a new log
I20260924 20:18:52.230902 17496 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260924 20:18:52.231142 17496 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:52.231179 17496 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7298a1ad7c34c70b5ad976e47bb846f, State: Initialized, Role: FOLLOWER
I20260924 20:18:52.231690 17496 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260924 20:18:52.231827 17496 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:52.231922 17496 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:52.232033 17496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:52.233029 17496 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260924 20:18:52.233393 17496 leader_election.cc:302] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7298a1ad7c34c70b5ad976e47bb846f; no voters:
I20260924 20:18:52.233734 17496 leader_election.cc:288] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:52.233922 17506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:52.234158 17506 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [term 1 LEADER]: Becoming Leader. State: Replica: d7298a1ad7c34c70b5ad976e47bb846f, State: Running, Role: LEADER
I20260924 20:18:52.234532 17506 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260924 20:18:52.235174 17496 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:52.236524 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7298a1ad7c34c70b5ad976e47bb846f. Latest consensus state: current_term: 1 leader_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } } }
I20260924 20:18:52.236534 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7298a1ad7c34c70b5ad976e47bb846f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } } }
I20260924 20:18:52.236675 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:52.236673 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:52.237100 17516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:52.239853 17516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:52.245261 17516 catalog_manager.cc:1347] Generated new cluster ID: 2de3745b018c4b6aaf89e3d3d00ccf80
I20260924 20:18:52.245333 17516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:52.255096 17516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:52.255425 17516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:52.274293 17516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7298a1ad7c34c70b5ad976e47bb846f: Generated new TSK 0
I20260924 20:18:52.275053 17516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:52.280262 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17428
2026-09-24T20:18:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:52 dist-test-slave-52fr krb5kdc[17527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:52 dist-test-slave-52fr krb5kdc[17527](info): set up 2 sockets
Sep 24 20:18:52 dist-test-slave-52fr krb5kdc[17527](info): commencing operation
krb5kdc: starting...
W20260924 20:18:53.314311 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:53.445163 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:53.564939 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:53.565295 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:53.565379 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:53.569684 17540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:53.569836 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:53.569883 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:53.569916 17540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:53.569952 17540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:53.575623 17540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:53.577239 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:53.578779 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:53.588404 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:53.588416 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:53.588440 17540 server_base.cc:1034] running on GCE node
W20260924 20:18:53.588406 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:53.589417 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:53.590770 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:53.591940 17540 hybrid_clock.cc:648] HybridClock initialized: now 1790281133591927 us; error 29 us; skew 500 ppm
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:53.595104 17540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:53.596140 17540 webserver.cc:460] Webserver started at http://127.0.0.1:33623/ using document root <none> and password file <none>
I20260924 20:18:53.596760 17540 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:53.596858 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:53.597097 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:53.599294 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "400ab996f4ac47fb8889a55ec9d62293"
format_stamp: "Formatted at 2026-09-24 20:18:53 on dist-test-slave-52fr"
server_key: "4a522741dbd35c3407ad737093bc81ad"
server_key_iv: "e1f4402e2cded1b95230d6d05c01b8a6"
server_key_version: "encryptionkey@0"
I20260924 20:18:53.599901 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "400ab996f4ac47fb8889a55ec9d62293"
format_stamp: "Formatted at 2026-09-24 20:18:53 on dist-test-slave-52fr"
server_key: "4a522741dbd35c3407ad737093bc81ad"
server_key_iv: "e1f4402e2cded1b95230d6d05c01b8a6"
server_key_version: "encryptionkey@0"
I20260924 20:18:53.603919 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:18:53.606787 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:53.608270 17540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:53.608448 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "400ab996f4ac47fb8889a55ec9d62293"
format_stamp: "Formatted at 2026-09-24 20:18:53 on dist-test-slave-52fr"
server_key: "4a522741dbd35c3407ad737093bc81ad"
server_key_iv: "e1f4402e2cded1b95230d6d05c01b8a6"
server_key_version: "encryptionkey@0"
I20260924 20:18:53.608583 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:53.640084 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:53.643801 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:53.644037 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:53.651301 17540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38673
I20260924 20:18:53.651326 17607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38673 every 8 connection(s)
I20260924 20:18:53.652248 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:53.652575 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17540
I20260924 20:18:53.652746 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:53.653029 30030 external_mini_cluster.cc:1428] Setting key 60780d6bf1f9761e2d87595ab996ab87
I20260924 20:18:53.655200 17608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:53.660717 17608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: Bootstrap starting.
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:18:53.663057 17608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:53.663904 17608 log.cc:824] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:53.665951 17608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: No bootstrap required, opened a new log
I20260924 20:18:53.667625 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:53.654560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42129 (local address 127.0.0.1:38673)
0924 20:18:53.654960 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:53.654981 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:53.655015 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:18:53.655705 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:53.655723 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:53.656063 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:53.656405 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:53.657194 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:53.658266 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:53.659132 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:53.659599 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:53.659789 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:53.662691 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:53.662697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:53.662712 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:53.662743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:53.664782 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:53.665221 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:53.665227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:53.665233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:53.665350 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:53.665645 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:53.665648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:53.665650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:53.666174 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:53.666372 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:53.666656 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 20:18:53.666950 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":146,"threads_started":1}
I20260924 20:18:53.669974 17608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } }
I20260924 20:18:53.670295 17608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:53.670349 17608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400ab996f4ac47fb8889a55ec9d62293, State: Initialized, Role: FOLLOWER
I20260924 20:18:53.670773 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:53.670807 17608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } }
I20260924 20:18:53.670984 17608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:53.671049 17608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:53.671147 17608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:53.672130 17608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } }
I20260924 20:18:53.672497 17608 leader_election.cc:302] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 400ab996f4ac47fb8889a55ec9d62293; no voters:
I20260924 20:18:53.672798 17608 leader_election.cc:288] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:53.672950 17617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:53.673308 17617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [term 1 LEADER]: Becoming Leader. State: Replica: 400ab996f4ac47fb8889a55ec9d62293, State: Running, Role: LEADER
I20260924 20:18:53.673717 17617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } }
I20260924 20:18:53.674137 17608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:53.675567 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 400ab996f4ac47fb8889a55ec9d62293. Latest consensus state: current_term: 1 leader_uuid: "400ab996f4ac47fb8889a55ec9d62293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } } }
I20260924 20:18:53.675705 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:53.676030 17618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "400ab996f4ac47fb8889a55ec9d62293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400ab996f4ac47fb8889a55ec9d62293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38673 } } }
I20260924 20:18:53.676127 17618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293 [sys.catalog]: This master's current role is: LEADER
W20260924 20:18:53.679471 17633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:18:53.679576 17633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:18:53.679728 17630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:53.680622 17630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:53.682389 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:53.673362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36049 (local address 127.0.0.1:38673)
0924 20:18:53.673506 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:53.673512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:53.673795 (+ 283us) server_negotiation.cc:407] Connection header received
0924 20:18:53.674001 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:53.674004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:53.674070 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:53.674162 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:53.675116 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:53.675958 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:53.677082 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:53.677415 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:53.677521 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:53.679141 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:53.679146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:53.679152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:53.679177 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:53.680679 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:53.681119 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:53.681123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:53.681125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:53.681209 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:53.681491 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:53.681495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:53.681496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:53.681681 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:53.681845 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:53.682098 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 20:18:53.682216 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:18:53.687692 17630 catalog_manager.cc:1347] Generated new cluster ID: 0d40d65bdf864a42984579c6c5823aaa
I20260924 20:18:53.687777 17630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:53.700487 17630 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:53.701927 17630 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:53.710070 17630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 400ab996f4ac47fb8889a55ec9d62293: Generated new TSK 0
I20260924 20:18:53.710968 17630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:53.714987 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17540
2026-09-24T20:18:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17639](info): set up 2 sockets
Sep 24 20:18:53 dist-test-slave-52fr krb5kdc[17639](info): commencing operation
krb5kdc: starting...
W20260924 20:18:54.754076 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:54 dist-test-slave-52fr krb5kdc[17639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:54.880702 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:54.995805 17652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:54.996163 17652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:54.996260 17652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:55.000547 17652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:55.000667 17652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:55.000707 17652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:55.000765 17652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:55.000804 17652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:55.006554 17652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:55.007758 17652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:55.008955 17652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:55.017984 17658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:55.018052 17660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:55.017984 17657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:55.018304 17652 server_base.cc:1034] running on GCE node
I20260924 20:18:55.018982 17652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:55.020146 17652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:55.021348 17652 hybrid_clock.cc:648] HybridClock initialized: now 1790281135021305 us; error 55 us; skew 500 ppm
Sep 24 20:18:55 dist-test-slave-52fr krb5kdc[17639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:55.024281 17652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:55.025207 17652 webserver.cc:460] Webserver started at http://127.0.0.1:40189/ using document root <none> and password file <none>
I20260924 20:18:55.025868 17652 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:55.025967 17652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:55.026247 17652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:55.028367 17652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "89e455551d8b4b5f860bf9cf4c47aa5a"
format_stamp: "Formatted at 2026-09-24 20:18:55 on dist-test-slave-52fr"
server_key: "26823d725fddb5a2c1e359b827939541"
server_key_iv: "a971dacf979c99a5909c072b781dc894"
server_key_version: "encryptionkey@0"
I20260924 20:18:55.028944 17652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "89e455551d8b4b5f860bf9cf4c47aa5a"
format_stamp: "Formatted at 2026-09-24 20:18:55 on dist-test-slave-52fr"
server_key: "26823d725fddb5a2c1e359b827939541"
server_key_iv: "a971dacf979c99a5909c072b781dc894"
server_key_version: "encryptionkey@0"
I20260924 20:18:55.032872 17652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:18:55.035521 17667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:55.036886 17652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:18:55.037058 17652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "89e455551d8b4b5f860bf9cf4c47aa5a"
format_stamp: "Formatted at 2026-09-24 20:18:55 on dist-test-slave-52fr"
server_key: "26823d725fddb5a2c1e359b827939541"
server_key_iv: "a971dacf979c99a5909c072b781dc894"
server_key_version: "encryptionkey@0"
I20260924 20:18:55.037184 17652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:55.054768 17652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:55.058359 17652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:55.058595 17652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:55.066051 17652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35861
I20260924 20:18:55.066076 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35861 every 8 connection(s)
I20260924 20:18:55.067063 17652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:55.067471 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17652
I20260924 20:18:55.067654 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:55.067983 30030 external_mini_cluster.cc:1428] Setting key 0ca8175875f79f88ebc973920db9bf6b
I20260924 20:18:55.070168 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:55.075767 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a: Bootstrap starting.
I20260924 20:18:55.077204 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:55.069487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36477 (local address 127.0.0.1:35861)
0924 20:18:55.069930 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:55.069952 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:55.069989 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:18:55.070698 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:55.070717 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:55.071066 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:55.071410 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:55.071831 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:55.072824 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:55.073748 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:55.074291 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:55.074467 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:55.075392 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:55.075397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:55.075410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:55.075791 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:55.075876 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:55.075910 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:55.076023 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:55.076140 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 20:18:55.076352 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":146,"threads_started":1}
I20260924 20:18:55.078493 17720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:55.079243 17720 log.cc:824] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:55.080534 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:55.081583 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a: No bootstrap required, opened a new log
I20260924 20:18:55.085619 17720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260924 20:18:55.085960 17720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:55.086036 17720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e455551d8b4b5f860bf9cf4c47aa5a, State: Initialized, Role: FOLLOWER
I20260924 20:18:55.086534 17720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260924 20:18:55.086700 17720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:55.086772 17720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:55.086871 17720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:55.087610 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:55.082813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58079 (local address 127.0.0.1:35861)
0924 20:18:55.082928 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:55.082934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:55.083068 (+ 134us) server_negotiation.cc:407] Connection header received
0924 20:18:55.083175 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:55.083177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:55.083215 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:55.083282 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:55.083569 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:55.084671 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:55.085838 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:55.086180 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:55.086293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:55.087151 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:55.087155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:55.087158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:55.087171 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:55.087200 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:55.087219 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:55.087304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:55.087427 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 20:18:55.087480 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:18:55.088112 17720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260924 20:18:55.088475 17720 leader_election.cc:302] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e455551d8b4b5f860bf9cf4c47aa5a; no voters:
I20260924 20:18:55.088768 17720 leader_election.cc:288] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:55.088907 17730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:55.089164 17730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [term 1 LEADER]: Becoming Leader. State: Replica: 89e455551d8b4b5f860bf9cf4c47aa5a, State: Running, Role: LEADER
I20260924 20:18:55.089510 17730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260924 20:18:55.090142 17720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:55.091436 17731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260924 20:18:55.091450 17732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e455551d8b4b5f860bf9cf4c47aa5a. Latest consensus state: current_term: 1 leader_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e455551d8b4b5f860bf9cf4c47aa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260924 20:18:55.091579 17731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:55.091583 17732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:55.091935 17740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:55.095232 17740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:55.101413 17740 catalog_manager.cc:1347] Generated new cluster ID: 65446c99dddd48aabe45cb7cb98402b1
I20260924 20:18:55.101513 17740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:55.130339 17740 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:55.131372 17740 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:55.142061 17740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e455551d8b4b5f860bf9cf4c47aa5a: Generated new TSK 0
I20260924 20:18:55.142818 17740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:55.147238 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17652
2026-09-24T20:18:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:55 dist-test-slave-52fr krb5kdc[17751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:55 dist-test-slave-52fr krb5kdc[17751](info): set up 2 sockets
Sep 24 20:18:55 dist-test-slave-52fr krb5kdc[17751](info): commencing operation
krb5kdc: starting...
W20260924 20:18:56.194115 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:56 dist-test-slave-52fr krb5kdc[17751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:56.322098 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:56.434974 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:56.435319 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:56.435416 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:56.439597 17764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:56.439702 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:56.439743 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:56.439778 17764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:56.439810 17764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:56.445338 17764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:56.446530 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:56.447618 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:56.456431 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:56.456453 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:56.456457 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:56.456403 17764 server_base.cc:1034] running on GCE node
I20260924 20:18:56.457320 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:56.458636 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:56.459856 17764 hybrid_clock.cc:648] HybridClock initialized: now 1790281136459823 us; error 52 us; skew 500 ppm
Sep 24 20:18:56 dist-test-slave-52fr krb5kdc[17751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:56.468438 17764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:56.469535 17764 webserver.cc:460] Webserver started at http://127.0.0.1:43033/ using document root <none> and password file <none>
I20260924 20:18:56.470232 17764 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:56.470333 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:56.470588 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:56.472859 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab"
format_stamp: "Formatted at 2026-09-24 20:18:56 on dist-test-slave-52fr"
server_key: "d61a44e2397ba26d13a2d258713df128"
server_key_iv: "18f4284cf3faa1faae300fd49a9160dc"
server_key_version: "encryptionkey@0"
I20260924 20:18:56.473498 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab"
format_stamp: "Formatted at 2026-09-24 20:18:56 on dist-test-slave-52fr"
server_key: "d61a44e2397ba26d13a2d258713df128"
server_key_iv: "18f4284cf3faa1faae300fd49a9160dc"
server_key_version: "encryptionkey@0"
I20260924 20:18:56.477845 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:56.480836 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:56.482502 17764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:56.482690 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab"
format_stamp: "Formatted at 2026-09-24 20:18:56 on dist-test-slave-52fr"
server_key: "d61a44e2397ba26d13a2d258713df128"
server_key_iv: "18f4284cf3faa1faae300fd49a9160dc"
server_key_version: "encryptionkey@0"
I20260924 20:18:56.482826 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:56.488097 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:56.492303 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:56.492560 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:56.500633 17764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46625
I20260924 20:18:56.500658 17831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46625 every 8 connection(s)
I20260924 20:18:56.501663 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:56.506170 17832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:56.509054 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17764
I20260924 20:18:56.509276 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:56.509605 30030 external_mini_cluster.cc:1428] Setting key fc306ec8135188473988f8725b17db02
I20260924 20:18:56.516686 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:56.511471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51099 (local address 127.0.0.1:46625)
0924 20:18:56.512045 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:56.512065 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:56.512111 (+ 46us) server_negotiation.cc:407] Connection header received
0924 20:18:56.513030 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:56.513060 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:56.513480 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:56.513895 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:56.514951 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:56.514959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:56.514978 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:56.514993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:56.515080 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:56.515121 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:56.515278 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:56.515564 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 20:18:56.515754 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":131,"threads_started":1}
I20260924 20:18:56.517323 17832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab: Bootstrap starting.
I20260924 20:18:56.520169 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:56.520906 17832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:56.521909 17832 log.cc:824] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:56.524433 17832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab: No bootstrap required, opened a new log
I20260924 20:18:56.524672 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:56.523096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41861 (local address 127.0.0.1:46625)
0924 20:18:56.523211 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:56.523217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:56.523233 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:18:56.523273 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:56.523276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:56.523315 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:56.523388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:56.524209 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:56.524212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:56.524215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:56.524230 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:56.524261 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:56.524275 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:56.524369 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:56.524481 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 20:18:56.524521 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:18:56.529021 17832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260924 20:18:56.529300 17832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:56.529371 17832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a43785b4fc5494d9f6ea6b5a1b195ab, State: Initialized, Role: FOLLOWER
I20260924 20:18:56.529897 17832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260924 20:18:56.530066 17832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:56.530143 17832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:56.530258 17832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:56.531566 17832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260924 20:18:56.531985 17832 leader_election.cc:302] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a43785b4fc5494d9f6ea6b5a1b195ab; no voters:
I20260924 20:18:56.532332 17832 leader_election.cc:288] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:56.532490 17842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:56.532711 17842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [term 1 LEADER]: Becoming Leader. State: Replica: 6a43785b4fc5494d9f6ea6b5a1b195ab, State: Running, Role: LEADER
I20260924 20:18:56.533020 17842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260924 20:18:56.533876 17832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:56.534420 17844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a43785b4fc5494d9f6ea6b5a1b195ab. Latest consensus state: current_term: 1 leader_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260924 20:18:56.534745 17844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:56.534816 17843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a43785b4fc5494d9f6ea6b5a1b195ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260924 20:18:56.534897 17843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:56.535421 17851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:56.538703 17851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:56.545061 17851 catalog_manager.cc:1347] Generated new cluster ID: 44340998a7e64eab861fb6bc6fcc8fcf
I20260924 20:18:56.545135 17851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:56.593487 17851 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:56.593891 17851 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:56.599100 17851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a43785b4fc5494d9f6ea6b5a1b195ab: Generated new TSK 0
I20260924 20:18:56.599875 17851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:56.607537 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17764
2026-09-24T20:18:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:56 dist-test-slave-52fr krb5kdc[17863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:56 dist-test-slave-52fr krb5kdc[17863](info): set up 2 sockets
Sep 24 20:18:56 dist-test-slave-52fr krb5kdc[17863](info): commencing operation
krb5kdc: starting...
W20260924 20:18:57.642118 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:57 dist-test-slave-52fr krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:57.765631 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:57.883002 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:57.883528 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:57.883677 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:57.887946 17876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:57.888024 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:57.888046 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:57.888063 17876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:57.888079 17876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:57.893946 17876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:57.895120 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:57.896229 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:57.905630 17884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:57.905643 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:57.905673 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:57.905728 17876 server_base.cc:1034] running on GCE node
I20260924 20:18:57.906582 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:57.907745 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:57.908963 17876 hybrid_clock.cc:648] HybridClock initialized: now 1790281137908907 us; error 66 us; skew 500 ppm
Sep 24 20:18:57 dist-test-slave-52fr krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:57.912278 17876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:57.913304 17876 webserver.cc:460] Webserver started at http://127.0.0.1:43325/ using document root <none> and password file <none>
I20260924 20:18:57.913983 17876 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:57.914077 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:57.914330 17876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:57.916822 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "6aa7074a59ea44569843b8479f1e9a30"
format_stamp: "Formatted at 2026-09-24 20:18:57 on dist-test-slave-52fr"
server_key: "db3b9e73058c45796d21bf7d0639eca8"
server_key_iv: "a8247402ae17ffcf27200699d6d64971"
server_key_version: "encryptionkey@0"
I20260924 20:18:57.917464 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "6aa7074a59ea44569843b8479f1e9a30"
format_stamp: "Formatted at 2026-09-24 20:18:57 on dist-test-slave-52fr"
server_key: "db3b9e73058c45796d21bf7d0639eca8"
server_key_iv: "a8247402ae17ffcf27200699d6d64971"
server_key_version: "encryptionkey@0"
I20260924 20:18:57.921963 17876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:18:57.924785 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:57.926465 17876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:57.926632 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "6aa7074a59ea44569843b8479f1e9a30"
format_stamp: "Formatted at 2026-09-24 20:18:57 on dist-test-slave-52fr"
server_key: "db3b9e73058c45796d21bf7d0639eca8"
server_key_iv: "a8247402ae17ffcf27200699d6d64971"
server_key_version: "encryptionkey@0"
I20260924 20:18:57.926785 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:57.942884 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:57.946455 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:57.946689 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:57.954499 17876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34593
I20260924 20:18:57.954536 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34593 every 8 connection(s)
I20260924 20:18:57.955516 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:57.958616 17944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:57.962651 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17876
I20260924 20:18:57.962813 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:57.963116 30030 external_mini_cluster.cc:1428] Setting key f111b4592fa66f53470b95572c13c682
I20260924 20:18:57.964888 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: Bootstrap starting.
I20260924 20:18:57.967517 17944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:57.968335 17944 log.cc:824] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:57.970741 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: No bootstrap required, opened a new log
Sep 24 20:18:57 dist-test-slave-52fr krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:18:57.975018 17944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260924 20:18:57.975345 17944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:57.975453 17944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa7074a59ea44569843b8479f1e9a30, State: Initialized, Role: FOLLOWER
I20260924 20:18:57.975991 17944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260924 20:18:57.976154 17944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:57.976246 17944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:57.976352 17944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:57.977313 17944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260924 20:18:57.977702 17944 leader_election.cc:302] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aa7074a59ea44569843b8479f1e9a30; no voters:
I20260924 20:18:57.978099 17944 leader_election.cc:288] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:57.978217 17949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:57.978407 17949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa7074a59ea44569843b8479f1e9a30, State: Running, Role: LEADER
I20260924 20:18:57.978770 17949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260924 20:18:57.979535 17944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:57.980232 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:57.964789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:43405 (local address 127.0.0.1:34593)
0924 20:18:57.965261 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:57.965277 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:57.965315 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:18:57.966057 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:57.966078 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:57.966457 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:57.966748 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:57.967773 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:57.969005 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:57.970277 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:57.970772 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:57.971021 (+ 249us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:57.974113 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:57.974119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:57.974135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:57.974197 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:57.976289 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:57.976978 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:57.976985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:57.976992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:57.977119 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:57.977499 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:57.977502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:57.977504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:57.978061 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:57.978304 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:57.979010 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 20:18:57.979212 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":147,"threads_started":1}
I20260924 20:18:57.980201 17951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aa7074a59ea44569843b8479f1e9a30. Latest consensus state: current_term: 1 leader_uuid: "6aa7074a59ea44569843b8479f1e9a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260924 20:18:57.980346 17951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:57.981290 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aa7074a59ea44569843b8479f1e9a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa7074a59ea44569843b8479f1e9a30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260924 20:18:57.981480 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:57.983641 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 20:18:57.985057 17964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:18:57.985134 17964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:18:57.985205 17956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:57.986101 17956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:57.993086 17956 catalog_manager.cc:1347] Generated new cluster ID: 8ba8999d1fad435fa7b6768ed0ce6ccb
I20260924 20:18:57.993149 17956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:57.994796 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:57.986207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:56347 (local address 127.0.0.1:34593)
0924 20:18:57.986312 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:57.986317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:57.986330 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:18:57.986369 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:57.986372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:57.986417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:57.986502 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:57.987315 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:57.988144 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:57.989229 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:57.989582 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:57.989698 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:57.991269 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:57.991273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:57.991275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:18:57.991299 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:57.992886 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:57.993416 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:57.993420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:57.993422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:57.993511 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:18:57.993884 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:18:57.993887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:18:57.993889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:18:57.994104 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:57.994252 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:57.994537 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 20:18:57.994638 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:18:58.006393 17956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:58.007385 17956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:58.019562 17956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aa7074a59ea44569843b8479f1e9a30: Generated new TSK 0
I20260924 20:18:58.020437 17956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:58.030869 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17876
W20260924 20:18:58.036549 17861 connection.cc:535] client connection to 127.0.0.1:34593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34593): Connection reset by peer (error 104)
2026-09-24T20:18:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:58 dist-test-slave-52fr krb5kdc[17975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:58 dist-test-slave-52fr krb5kdc[17975](info): set up 2 sockets
Sep 24 20:18:58 dist-test-slave-52fr krb5kdc[17975](info): commencing operation
krb5kdc: starting...
W20260924 20:18:59.074137 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:18:59 dist-test-slave-52fr krb5kdc[17975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:18:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:18:59.205008 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:18:59.318785 17988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:18:59.319207 17988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:18:59.319275 17988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:18:59.323478 17988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:18:59.323541 17988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:18:59.323561 17988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:18:59.323580 17988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:18:59.323594 17988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:18:59.329224 17988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:18:59.330412 17988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:18:59.331465 17988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:18:59.343116 17994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:59.343191 17996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:18:59.343113 17993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:18:59.343478 17988 server_base.cc:1034] running on GCE node
I20260924 20:18:59.344086 17988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:18:59.345260 17988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:18:59.346493 17988 hybrid_clock.cc:648] HybridClock initialized: now 1790281139346480 us; error 30 us; skew 500 ppm
Sep 24 20:18:59 dist-test-slave-52fr krb5kdc[17975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:18:59.349416 17988 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:18:59.350400 17988 webserver.cc:460] Webserver started at http://127.0.0.1:34059/ using document root <none> and password file <none>
I20260924 20:18:59.350967 17988 fs_manager.cc:359] Metadata directory not provided
I20260924 20:18:59.351029 17988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:18:59.351192 17988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:18:59.353267 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "9444c767ea9e4db49744f07732fa3b50"
format_stamp: "Formatted at 2026-09-24 20:18:59 on dist-test-slave-52fr"
server_key: "b402893bf91fdbb00889d77cc3593bb1"
server_key_iv: "8c49c78c7eab0362a86f94e3e5599a47"
server_key_version: "encryptionkey@0"
I20260924 20:18:59.353914 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "9444c767ea9e4db49744f07732fa3b50"
format_stamp: "Formatted at 2026-09-24 20:18:59 on dist-test-slave-52fr"
server_key: "b402893bf91fdbb00889d77cc3593bb1"
server_key_iv: "8c49c78c7eab0362a86f94e3e5599a47"
server_key_version: "encryptionkey@0"
I20260924 20:18:59.357657 17988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:18:59.360200 18003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:18:59.361426 17988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:18:59.361557 17988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "9444c767ea9e4db49744f07732fa3b50"
format_stamp: "Formatted at 2026-09-24 20:18:59 on dist-test-slave-52fr"
server_key: "b402893bf91fdbb00889d77cc3593bb1"
server_key_iv: "8c49c78c7eab0362a86f94e3e5599a47"
server_key_version: "encryptionkey@0"
I20260924 20:18:59.361670 17988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:18:59.375746 17988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:18:59.379163 17988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:18:59.379391 17988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:18:59.386685 17988 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37993
I20260924 20:18:59.386711 18055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37993 every 8 connection(s)
I20260924 20:18:59.387658 17988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:18:59.390763 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:18:59.392257 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 17988
I20260924 20:18:59.392426 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:18:59.392737 30030 external_mini_cluster.cc:1428] Setting key 9e28a311d335f19a22a3fd56e973119b
I20260924 20:18:59.397249 18056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50: Bootstrap starting.
I20260924 20:18:59.399628 18056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50: Neither blocks nor log segments found. Creating new log.
I20260924 20:18:59.400490 18056 log.cc:824] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50: Log is configured to *not* fsync() on all Append() calls
I20260924 20:18:59.402477 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:59.394445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:35551 (local address 127.0.0.1:37993)
0924 20:18:59.394910 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:59.394929 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:59.394963 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:18:59.395598 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:59.395615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:59.395959 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:59.396300 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:59.396710 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:59.397750 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:59.398713 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:59.399198 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:59.399389 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:59.400330 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:59.400335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:59.400374 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:59.400821 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:59.400914 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:59.400948 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:59.401087 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:59.401304 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 20:18:59.401483 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
I20260924 20:18:59.402912 18056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50: No bootstrap required, opened a new log
I20260924 20:18:59.405603 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:18:59.407044 18056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260924 20:18:59.407275 18056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:18:59.407332 18056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9444c767ea9e4db49744f07732fa3b50, State: Initialized, Role: FOLLOWER
I20260924 20:18:59.407825 18056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260924 20:18:59.408015 18056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:18:59.408102 18056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:18:59.408208 18056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:18:59.409195 18056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260924 20:18:59.409548 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9444c767ea9e4db49744f07732fa3b50; no voters:
I20260924 20:18:59.409932 18056 leader_election.cc:288] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:18:59.410000 18066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:18:59.410223 18066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [term 1 LEADER]: Becoming Leader. State: Replica: 9444c767ea9e4db49744f07732fa3b50, State: Running, Role: LEADER
I20260924 20:18:59.410573 18066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260924 20:18:59.411269 18056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:18:59.411767 18066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9444c767ea9e4db49744f07732fa3b50. Latest consensus state: current_term: 1 leader_uuid: "9444c767ea9e4db49744f07732fa3b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } } }
I20260924 20:18:59.411932 18066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:59.412331 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:18:59.407767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:38697 (local address 127.0.0.1:37993)
0924 20:18:59.407919 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:18:59.407924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:18:59.408083 (+ 159us) server_negotiation.cc:407] Connection header received
0924 20:18:59.408247 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:18:59.408251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:18:59.408320 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:18:59.408399 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:18:59.408675 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:59.409208 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:59.410283 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:18:59.410585 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:18:59.410646 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:18:59.411681 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:18:59.411686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:18:59.411688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:18:59.411703 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:18:59.411732 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:18:59.411745 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:18:59.411842 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 20:18:59.412090 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 20:18:59.412136 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:18:59.412365 18071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:18:59.412200 18067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9444c767ea9e4db49744f07732fa3b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9444c767ea9e4db49744f07732fa3b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } } }
I20260924 20:18:59.412953 18067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50 [sys.catalog]: This master's current role is: LEADER
I20260924 20:18:59.415648 18071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:18:59.421219 18071 catalog_manager.cc:1347] Generated new cluster ID: 93455eb3a2a74d9889f184a56d5f65f8
I20260924 20:18:59.421289 18071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:18:59.434265 18071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:18:59.435602 18071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:18:59.442425 18071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9444c767ea9e4db49744f07732fa3b50: Generated new TSK 0
I20260924 20:18:59.443244 18071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:18:59.454131 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 17988
W20260924 20:18:59.460001 17973 connection.cc:535] client connection to 127.0.0.1:37993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37993): Connection reset by peer (error 104)
2026-09-24T20:18:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:18:59 dist-test-slave-52fr krb5kdc[18087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:18:59 dist-test-slave-52fr krb5kdc[18087](info): set up 2 sockets
Sep 24 20:18:59 dist-test-slave-52fr krb5kdc[18087](info): commencing operation
krb5kdc: starting...
W20260924 20:19:00.506112 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:00 dist-test-slave-52fr krb5kdc[18087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:19:00.633263 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:00.749074 18100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:00.749404 18100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:00.749490 18100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:00.753605 18100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:00.753710 18100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:00.753750 18100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:00.753803 18100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:00.753890 18100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:00.759378 18100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:00.760481 18100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:00.761523 18100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:00.770401 18106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:00.770417 18105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:00.770417 18108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:00.770458 18100 server_base.cc:1034] running on GCE node
I20260924 20:19:00.771217 18100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:00.772341 18100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:00.773562 18100 hybrid_clock.cc:648] HybridClock initialized: now 1790281140773549 us; error 27 us; skew 500 ppm
Sep 24 20:19:00 dist-test-slave-52fr krb5kdc[18087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:00.776517 18100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:19:00.777417 18100 webserver.cc:460] Webserver started at http://127.0.0.1:38617/ using document root <none> and password file <none>
I20260924 20:19:00.778120 18100 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:00.778213 18100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:00.778439 18100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:00.780495 18100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d"
format_stamp: "Formatted at 2026-09-24 20:19:00 on dist-test-slave-52fr"
server_key: "e513d15f68c3663e1269449d509154bc"
server_key_iv: "c2a7245bf6c5f59c073e88d0694c59b1"
server_key_version: "encryptionkey@0"
I20260924 20:19:00.781064 18100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d"
format_stamp: "Formatted at 2026-09-24 20:19:00 on dist-test-slave-52fr"
server_key: "e513d15f68c3663e1269449d509154bc"
server_key_iv: "c2a7245bf6c5f59c073e88d0694c59b1"
server_key_version: "encryptionkey@0"
I20260924 20:19:00.784785 18100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:19:00.787245 18115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:00.788527 18100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:19:00.788666 18100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d"
format_stamp: "Formatted at 2026-09-24 20:19:00 on dist-test-slave-52fr"
server_key: "e513d15f68c3663e1269449d509154bc"
server_key_iv: "c2a7245bf6c5f59c073e88d0694c59b1"
server_key_version: "encryptionkey@0"
I20260924 20:19:00.788777 18100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:00.793354 18100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:00.796499 18100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:00.796692 18100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:00.803718 18100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260924 20:19:00.803781 18167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260924 20:19:00.804693 18100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:00.807567 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:00.808663 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18100
I20260924 20:19:00.808772 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:00.809105 30030 external_mini_cluster.cc:1428] Setting key cf39fb7542e94c1438436eb77abb7e96
I20260924 20:19:00.814208 18168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d: Bootstrap starting.
I20260924 20:19:00.814318 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:00.810473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:54677 (local address 127.0.0.1:43105)
0924 20:19:00.810861 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:00.810886 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:00.810923 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:19:00.811545 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:00.811560 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:00.811841 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:00.812122 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:00.812959 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:00.812964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:00.812975 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:00.812985 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:00.813036 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:00.813055 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:00.813167 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:00.813299 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 20:19:00.813525 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260924 20:19:00.816880 18168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:00.817327 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:00.817746 18168 log.cc:824] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:00.819872 18168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d: No bootstrap required, opened a new log
I20260924 20:19:00.822065 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:00.820365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:33323 (local address 127.0.0.1:43105)
0924 20:19:00.820496 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:00.820501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:00.820709 (+ 208us) server_negotiation.cc:407] Connection header received
0924 20:19:00.820843 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:00.820845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:00.820875 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:00.820934 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:00.821630 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:00.821633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:00.821635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:00.821646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:00.821670 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:00.821680 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:00.821748 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:00.821874 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 20:19:00.821908 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:19:00.824395 18168 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 20:19:00.824687 18168 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:00.824806 18168 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8480a705fd438da0c1aa2bc8d5ec4d, State: Initialized, Role: FOLLOWER
I20260924 20:19:00.825304 18168 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 20:19:00.825456 18168 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:00.825536 18168 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:00.825634 18168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:00.826606 18168 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 20:19:00.826960 18168 leader_election.cc:302] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8480a705fd438da0c1aa2bc8d5ec4d; no voters:
I20260924 20:19:00.827308 18168 leader_election.cc:288] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:00.827380 18178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:00.827566 18178 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [term 1 LEADER]: Becoming Leader. State: Replica: fc8480a705fd438da0c1aa2bc8d5ec4d, State: Running, Role: LEADER
I20260924 20:19:00.827962 18178 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 20:19:00.828723 18168 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:00.829859 18180 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260924 20:19:00.829893 18179 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8480a705fd438da0c1aa2bc8d5ec4d. Latest consensus state: current_term: 1 leader_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8480a705fd438da0c1aa2bc8d5ec4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260924 20:19:00.830097 18180 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:00.830102 18179 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:00.830550 18187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:00.833986 18187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:00.839800 18187 catalog_manager.cc:1347] Generated new cluster ID: d40ced1872ce402e90a0ad12c5d3cc54
I20260924 20:19:00.839897 18187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:00.853488 18187 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:00.853888 18187 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:00.868649 18187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8480a705fd438da0c1aa2bc8d5ec4d: Generated new TSK 0
I20260924 20:19:00.869434 18187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:00.873919 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18100
2026-09-24T20:19:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:00 dist-test-slave-52fr krb5kdc[18199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:00 dist-test-slave-52fr krb5kdc[18199](info): set up 2 sockets
Sep 24 20:19:00 dist-test-slave-52fr krb5kdc[18199](info): commencing operation
krb5kdc: starting...
W20260924 20:19:01.910420 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:01 dist-test-slave-52fr krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:19:02.035385 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:02.148944 18212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:02.149281 18212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:02.149374 18212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:02.153573 18212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:02.153687 18212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:02.153726 18212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:02.153811 18212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:02.153872 18212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:02.159363 18212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:02.160527 18212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:02.161633 18212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:19:02.170394 18212 server_base.cc:1034] running on GCE node
W20260924 20:19:02.170408 18217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:02.170559 18220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:02.170414 18218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:02.171231 18212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:02.172367 18212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:02.173535 18212 hybrid_clock.cc:648] HybridClock initialized: now 1790281142173518 us; error 33 us; skew 500 ppm
Sep 24 20:19:02 dist-test-slave-52fr krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:02.180198 18212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:19:02.181176 18212 webserver.cc:460] Webserver started at http://127.0.0.1:46095/ using document root <none> and password file <none>
I20260924 20:19:02.181806 18212 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:02.181906 18212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:02.182147 18212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:02.184505 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "5c1494b589a042e99ac86a68911a2ecc"
format_stamp: "Formatted at 2026-09-24 20:19:02 on dist-test-slave-52fr"
server_key: "734b7b3d03d00edd598d94991a971be9"
server_key_iv: "30576efcce87e4fddc2e8cf7089b1906"
server_key_version: "encryptionkey@0"
I20260924 20:19:02.185210 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "5c1494b589a042e99ac86a68911a2ecc"
format_stamp: "Formatted at 2026-09-24 20:19:02 on dist-test-slave-52fr"
server_key: "734b7b3d03d00edd598d94991a971be9"
server_key_iv: "30576efcce87e4fddc2e8cf7089b1906"
server_key_version: "encryptionkey@0"
I20260924 20:19:02.189237 18212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:02.191679 18227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:02.192901 18212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:19:02.193039 18212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5c1494b589a042e99ac86a68911a2ecc"
format_stamp: "Formatted at 2026-09-24 20:19:02 on dist-test-slave-52fr"
server_key: "734b7b3d03d00edd598d94991a971be9"
server_key_iv: "30576efcce87e4fddc2e8cf7089b1906"
server_key_version: "encryptionkey@0"
I20260924 20:19:02.193159 18212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:02.210724 18212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:02.214123 18212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:02.214349 18212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:02.221323 18212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45989
I20260924 20:19:02.221438 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45989 every 8 connection(s)
I20260924 20:19:02.222394 18212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:02.225271 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:02.231007 18280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc: Bootstrap starting.
I20260924 20:19:02.232339 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18212
I20260924 20:19:02.232506 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:02.232777 30030 external_mini_cluster.cc:1428] Setting key 5961511729fa24f773a7beb330bd31c3
I20260924 20:19:02.233632 18280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:02.234588 18280 log.cc:824] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:02.236748 18280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc: No bootstrap required, opened a new log
I20260924 20:19:02.240998 18280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260924 20:19:02.241253 18280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:02.241334 18280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1494b589a042e99ac86a68911a2ecc, State: Initialized, Role: FOLLOWER
I20260924 20:19:02.241833 18280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260924 20:19:02.242030 18280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:02.242121 18280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:02.242226 18280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 0 FOLLOWER]: Advancing to term 1
Sep 24 20:19:02 dist-test-slave-52fr krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:19:02.243324 18280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260924 20:19:02.243718 18280 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1494b589a042e99ac86a68911a2ecc; no voters:
I20260924 20:19:02.244120 18280 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:02.244230 18285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:02.244453 18285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [term 1 LEADER]: Becoming Leader. State: Replica: 5c1494b589a042e99ac86a68911a2ecc, State: Running, Role: LEADER
I20260924 20:19:02.244836 18285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260924 20:19:02.245551 18280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:02.246474 18286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1494b589a042e99ac86a68911a2ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260924 20:19:02.246649 18286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:02.248034 18287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1494b589a042e99ac86a68911a2ecc. Latest consensus state: current_term: 1 leader_uuid: "5c1494b589a042e99ac86a68911a2ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1494b589a042e99ac86a68911a2ecc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260924 20:19:02.248204 18294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:02.248219 18287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:02.249289 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:02.233986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:36227 (local address 127.0.0.1:45989)
0924 20:19:02.234499 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:02.234518 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:02.234559 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:19:02.235364 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:02.235388 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:02.235802 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:02.236156 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:02.236858 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:02.238386 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:02.239304 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:02.240110 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:02.240274 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:02.243206 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:02.243215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:02.243236 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:02.243277 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:02.245306 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:02.245972 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:02.245980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:02.245989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:02.246138 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:02.246618 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:02.246623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:02.246625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:02.247160 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:02.247426 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:02.248233 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 20:19:02.248496 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":123,"threads_started":1}
I20260924 20:19:02.251538 18294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:02.252208 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:02.257539 18294 catalog_manager.cc:1347] Generated new cluster ID: 58dd535cc836483fa122d206e9b7c586
I20260924 20:19:02.257601 18294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:02.263520 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:02.254232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:46693 (local address 127.0.0.1:45989)
0924 20:19:02.254330 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:02.254333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:02.254835 (+ 502us) server_negotiation.cc:407] Connection header received
0924 20:19:02.254953 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:02.254956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:02.255004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:02.255086 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:02.255880 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:02.256787 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:02.257866 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:02.258228 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:02.258346 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:02.259933 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:02.259937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:02.259940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:02.259964 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:02.261667 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:02.262197 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:02.262207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:02.262209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:02.262282 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:02.262648 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:02.262651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:02.262652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:02.262807 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:02.262967 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:02.263230 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 20:19:02.263323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260924 20:19:02.264762 18294 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:02.266090 18294 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:02.274507 18294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1494b589a042e99ac86a68911a2ecc: Generated new TSK 0
I20260924 20:19:02.275319 18294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:02.278800 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18212
W20260924 20:19:02.284719 18197 connection.cc:535] client connection to 127.0.0.1:45989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45989): Connection reset by peer (error 104)
2026-09-24T20:19:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:02 dist-test-slave-52fr krb5kdc[18311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:02 dist-test-slave-52fr krb5kdc[18311](info): set up 2 sockets
Sep 24 20:19:02 dist-test-slave-52fr krb5kdc[18311](info): commencing operation
krb5kdc: starting...
W20260924 20:19:03.321940 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:03 dist-test-slave-52fr krb5kdc[18311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:19:03.452850 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:03.570272 18324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:03.570575 18324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:03.570623 18324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:03.577474 18324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:03.577555 18324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:03.577575 18324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:03.577593 18324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:03.577638 18324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:03.584153 18324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:03.585489 18324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:03.586632 18324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:03.596226 18329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:03.596230 18332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:03.596379 18324 server_base.cc:1034] running on GCE node
W20260924 20:19:03.596251 18330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:03.597224 18324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:03.598946 18324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:03.600188 18324 hybrid_clock.cc:648] HybridClock initialized: now 1790281143600173 us; error 32 us; skew 500 ppm
Sep 24 20:19:03 dist-test-slave-52fr krb5kdc[18311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:03.603259 18324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:19:03.604246 18324 webserver.cc:460] Webserver started at http://127.0.0.1:40833/ using document root <none> and password file <none>
I20260924 20:19:03.604791 18324 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:03.604856 18324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:03.605026 18324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:03.607192 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "135010ce9a7248538017b2ac1dd85228"
format_stamp: "Formatted at 2026-09-24 20:19:03 on dist-test-slave-52fr"
server_key: "df40765672e0f69c668966d27c9a0eaa"
server_key_iv: "bfe4768c9732a0bfdcbaa3dc98d3a758"
server_key_version: "encryptionkey@0"
I20260924 20:19:03.607728 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "135010ce9a7248538017b2ac1dd85228"
format_stamp: "Formatted at 2026-09-24 20:19:03 on dist-test-slave-52fr"
server_key: "df40765672e0f69c668966d27c9a0eaa"
server_key_iv: "bfe4768c9732a0bfdcbaa3dc98d3a758"
server_key_version: "encryptionkey@0"
I20260924 20:19:03.611742 18324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:19:03.614323 18339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:03.615645 18324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:03.615756 18324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "135010ce9a7248538017b2ac1dd85228"
format_stamp: "Formatted at 2026-09-24 20:19:03 on dist-test-slave-52fr"
server_key: "df40765672e0f69c668966d27c9a0eaa"
server_key_iv: "bfe4768c9732a0bfdcbaa3dc98d3a758"
server_key_version: "encryptionkey@0"
I20260924 20:19:03.615849 18324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:03.658967 18324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:03.662453 18324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:03.662611 18324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:03.669819 18324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35243
I20260924 20:19:03.669847 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35243 every 8 connection(s)
I20260924 20:19:03.670778 18324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:03.673525 18392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:03.679388 18392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228: Bootstrap starting.
I20260924 20:19:03.680577 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18324
I20260924 20:19:03.680737 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:03.681074 30030 external_mini_cluster.cc:1428] Setting key f56a5c7c58cadcb64ca34cf856b02480
I20260924 20:19:03.682507 18392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:03.683319 18392 log.cc:824] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:03.685513 18392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228: No bootstrap required, opened a new log
I20260924 20:19:03.689270 18392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 20:19:03.689550 18392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:03.689662 18392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135010ce9a7248538017b2ac1dd85228, State: Initialized, Role: FOLLOWER
I20260924 20:19:03.690196 18392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 20:19:03.690358 18392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:03.690448 18392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:03.690549 18392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:03.690618 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:03.682322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:47627 (local address 127.0.0.1:35243)
0924 20:19:03.682823 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:03.682839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:03.682876 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:19:03.683644 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:03.683685 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:03.684149 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:03.684517 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:03.684949 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:03.686059 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:03.686997 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:03.687494 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:03.687667 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:03.688705 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:03.688710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:03.688724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:03.689123 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:03.689177 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:03.689199 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:03.689354 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:03.689543 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:19:03.689687 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":153,"threads_started":1}
I20260924 20:19:03.691731 18392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 20:19:03.692095 18392 leader_election.cc:302] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135010ce9a7248538017b2ac1dd85228; no voters:
I20260924 20:19:03.692406 18392 leader_election.cc:288] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:03.692507 18397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:03.692768 18397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [term 1 LEADER]: Becoming Leader. State: Replica: 135010ce9a7248538017b2ac1dd85228, State: Running, Role: LEADER
I20260924 20:19:03.693111 18397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 20:19:03.693714 18392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:03.693974 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:03.695030 18399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135010ce9a7248538017b2ac1dd85228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 20:19:03.695096 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135010ce9a7248538017b2ac1dd85228. Latest consensus state: current_term: 1 leader_uuid: "135010ce9a7248538017b2ac1dd85228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135010ce9a7248538017b2ac1dd85228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 20:19:03.695164 18399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:03.695164 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:03.695557 18409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:03.698438 18409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:03.704097 18409 catalog_manager.cc:1347] Generated new cluster ID: 5bab79e1f3014c20b128942065449e02
I20260924 20:19:03.704236 18409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:03.704384 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:03.696800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:59131 (local address 127.0.0.1:35243)
0924 20:19:03.696897 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:03.696901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:03.700184 (+ 3283us) server_negotiation.cc:407] Connection header received
0924 20:19:03.700352 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:03.700355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:03.700404 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:03.700488 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:03.700751 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:03.701280 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:03.702268 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:03.702632 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:03.702743 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:03.703712 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:03.703718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:03.703722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:03.703740 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:03.703777 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:03.703794 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:03.703901 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:03.704160 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 20:19:03.704198 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:19:03.717598 18409 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:03.718572 18409 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:03.724644 18409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135010ce9a7248538017b2ac1dd85228: Generated new TSK 0
I20260924 20:19:03.725539 18409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:03.735440 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18324
W20260924 20:19:03.740598 18309 connection.cc:535] client connection to 127.0.0.1:35243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35243): Connection reset by peer (error 104)
2026-09-24T20:19:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:03 dist-test-slave-52fr krb5kdc[18423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:03 dist-test-slave-52fr krb5kdc[18423](info): set up 2 sockets
Sep 24 20:19:03 dist-test-slave-52fr krb5kdc[18423](info): commencing operation
krb5kdc: starting...
W20260924 20:19:04.774143 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:04 dist-test-slave-52fr krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:19:04.901264 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:05.016433 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:05.016729 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:05.016773 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:05.020931 18436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:05.021010 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:05.021030 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:05.021049 18436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:05.021064 18436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:05.026593 18436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:05.027810 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:05.028962 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:05.038028 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:05.038101 18436 server_base.cc:1034] running on GCE node
W20260924 20:19:05.038039 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:05.038039 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:05.038899 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:05.040123 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:05.041348 18436 hybrid_clock.cc:648] HybridClock initialized: now 1790281145041319 us; error 51 us; skew 500 ppm
Sep 24 20:19:05 dist-test-slave-52fr krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:05.044761 18436 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:19:05.045711 18436 webserver.cc:460] Webserver started at http://127.0.0.1:45925/ using document root <none> and password file <none>
I20260924 20:19:05.046340 18436 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:05.046408 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:05.046661 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:05.048794 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "aeee761dedb7491ca48828952b9e6cdd"
format_stamp: "Formatted at 2026-09-24 20:19:05 on dist-test-slave-52fr"
server_key: "581160680ccc7200a82b6b1daa7597d1"
server_key_iv: "957a6ae88dacc27be42bd83d9838d505"
server_key_version: "encryptionkey@0"
I20260924 20:19:05.049387 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "aeee761dedb7491ca48828952b9e6cdd"
format_stamp: "Formatted at 2026-09-24 20:19:05 on dist-test-slave-52fr"
server_key: "581160680ccc7200a82b6b1daa7597d1"
server_key_iv: "957a6ae88dacc27be42bd83d9838d505"
server_key_version: "encryptionkey@0"
I20260924 20:19:05.053459 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:19:05.056348 18451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:05.057684 18436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 20:19:05.057873 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "aeee761dedb7491ca48828952b9e6cdd"
format_stamp: "Formatted at 2026-09-24 20:19:05 on dist-test-slave-52fr"
server_key: "581160680ccc7200a82b6b1daa7597d1"
server_key_iv: "957a6ae88dacc27be42bd83d9838d505"
server_key_version: "encryptionkey@0"
I20260924 20:19:05.058002 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:05.062728 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:05.065686 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:05.065923 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:05.073194 18436 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42795
I20260924 20:19:05.073194 18503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42795 every 8 connection(s)
I20260924 20:19:05.074373 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:05.077235 18504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:05.078061 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18436
I20260924 20:19:05.078219 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:05.078521 30030 external_mini_cluster.cc:1428] Setting key 723b4a4226e6582a82014137805fbdfb
I20260924 20:19:05.083938 18504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd: Bootstrap starting.
I20260924 20:19:05.083976 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:05.080010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:50481 (local address 127.0.0.1:42795)
0924 20:19:05.080502 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:05.080516 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:05.080553 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:19:05.081201 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:05.081219 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:05.081532 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:05.081913 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:05.082748 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:05.082752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:05.082765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:05.082775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:05.082831 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:05.082851 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:05.083008 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:05.083081 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 20:19:05.083251 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":149,"threads_started":1}
I20260924 20:19:05.086596 18504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:05.086936 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:05.087450 18504 log.cc:824] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:05.089968 18504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd: No bootstrap required, opened a new log
I20260924 20:19:05.090939 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:05.089098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:40853 (local address 127.0.0.1:42795)
0924 20:19:05.089232 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:05.089237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:05.089387 (+ 150us) server_negotiation.cc:407] Connection header received
0924 20:19:05.089479 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:05.089482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:05.089519 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:05.089613 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:05.090557 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:05.090561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:05.090563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:05.090576 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:05.090606 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:05.090619 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:05.090706 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:05.090769 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 20:19:05.090809 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:19:05.094255 18504 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } }
I20260924 20:19:05.094540 18504 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:05.094586 18504 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeee761dedb7491ca48828952b9e6cdd, State: Initialized, Role: FOLLOWER
I20260924 20:19:05.095114 18504 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } }
I20260924 20:19:05.095265 18504 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:05.095371 18504 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:05.095490 18504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:05.096539 18504 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } }
I20260924 20:19:05.096920 18504 leader_election.cc:302] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeee761dedb7491ca48828952b9e6cdd; no voters:
I20260924 20:19:05.097261 18504 leader_election.cc:288] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:05.097556 18514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:05.097803 18514 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [term 1 LEADER]: Becoming Leader. State: Replica: aeee761dedb7491ca48828952b9e6cdd, State: Running, Role: LEADER
I20260924 20:19:05.098148 18514 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } }
I20260924 20:19:05.098660 18504 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:05.099989 18515 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeee761dedb7491ca48828952b9e6cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } } }
I20260924 20:19:05.100040 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeee761dedb7491ca48828952b9e6cdd. Latest consensus state: current_term: 1 leader_uuid: "aeee761dedb7491ca48828952b9e6cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeee761dedb7491ca48828952b9e6cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42795 } } }
I20260924 20:19:05.100119 18515 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:05.100149 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:05.100530 18523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:05.103850 18523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:05.108959 18523 catalog_manager.cc:1347] Generated new cluster ID: b423e7ed96984d54823ac6bbd847b812
I20260924 20:19:05.109025 18523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:05.120554 18523 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:05.120885 18523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:05.129828 18523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeee761dedb7491ca48828952b9e6cdd: Generated new TSK 0
I20260924 20:19:05.130656 18523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:05.135761 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18436
2026-09-24T20:19:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:05 dist-test-slave-52fr krb5kdc[18535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:05 dist-test-slave-52fr krb5kdc[18535](info): set up 2 sockets
Sep 24 20:19:05 dist-test-slave-52fr krb5kdc[18535](info): commencing operation
krb5kdc: starting...
W20260924 20:19:06.186174 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:06.309715 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41401
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:06.432548 18548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:06.432849 18548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:06.432893 18548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:06.437127 18548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:06.437199 18548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:06.437219 18548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:06.437237 18548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:06.437252 18548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:06.442835 18548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:41401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:06.443928 18548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:06.445020 18548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:06.454438 18553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:06.454418 18554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:06.454418 18556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:06.454528 18548 server_base.cc:1034] running on GCE node
I20260924 20:19:06.455443 18548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:06.456596 18548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:06.457839 18548 hybrid_clock.cc:648] HybridClock initialized: now 1790281146457806 us; error 31 us; skew 500 ppm
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:06.465090 18548 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:06.466378 18548 webserver.cc:460] Webserver started at http://127.29.83.190:38183/ using document root <none> and password file <none>
I20260924 20:19:06.466992 18548 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:06.467062 18548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:06.467293 18548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:06.469512 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "1e67f2970a544cbc88c3c9cfa730db0e"
format_stamp: "Formatted at 2026-09-24 20:19:06 on dist-test-slave-52fr"
server_key: "5e2acdd125b19e10483ad12404735d41"
server_key_iv: "f359dd4c86e7876936ee5afac4c65113"
server_key_version: "encryptionkey@0"
I20260924 20:19:06.470197 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "1e67f2970a544cbc88c3c9cfa730db0e"
format_stamp: "Formatted at 2026-09-24 20:19:06 on dist-test-slave-52fr"
server_key: "5e2acdd125b19e10483ad12404735d41"
server_key_iv: "f359dd4c86e7876936ee5afac4c65113"
server_key_version: "encryptionkey@0"
I20260924 20:19:06.474706 18548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:19:06.477403 18563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:06.478940 18548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:06.479130 18548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "1e67f2970a544cbc88c3c9cfa730db0e"
format_stamp: "Formatted at 2026-09-24 20:19:06 on dist-test-slave-52fr"
server_key: "5e2acdd125b19e10483ad12404735d41"
server_key_iv: "f359dd4c86e7876936ee5afac4c65113"
server_key_version: "encryptionkey@0"
I20260924 20:19:06.479271 18548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:06.496095 18548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:06.499812 18548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:06.500051 18548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:06.508065 18548 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:41401
I20260924 20:19:06.508134 18615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:41401 every 8 connection(s)
I20260924 20:19:06.509119 18548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:06.512183 18616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:06.516664 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18548
I20260924 20:19:06.516821 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:06.517115 30030 external_mini_cluster.cc:1428] Setting key 7400e7fb0f9bb43a6210fb0e2e59776b
I20260924 20:19:06.518325 18616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: Bootstrap starting.
I20260924 20:19:06.520656 18616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:06.521512 18616 log.cc:824] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:06.523727 18616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: No bootstrap required, opened a new log
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:06.528530 18616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } }
I20260924 20:19:06.528894 18616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:06.528952 18616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e67f2970a544cbc88c3c9cfa730db0e, State: Initialized, Role: FOLLOWER
I20260924 20:19:06.529480 18616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } }
I20260924 20:19:06.529659 18616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:06.529749 18616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:06.529891 18616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:06.531170 18616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } }
I20260924 20:19:06.531551 18616 leader_election.cc:302] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e67f2970a544cbc88c3c9cfa730db0e; no voters:
I20260924 20:19:06.531989 18616 leader_election.cc:288] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:06.532186 18621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:06.532419 18621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [term 1 LEADER]: Becoming Leader. State: Replica: 1e67f2970a544cbc88c3c9cfa730db0e, State: Running, Role: LEADER
I20260924 20:19:06.532778 18621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } }
I20260924 20:19:06.533183 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:06.518638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43705 (local address 127.29.83.190:41401)
0924 20:19:06.519051 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:06.519067 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:06.519111 (+ 44us) server_negotiation.cc:407] Connection header received
0924 20:19:06.519741 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:06.519758 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:06.520092 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:06.520378 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:06.521284 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:06.522401 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:06.523572 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:06.524102 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:06.524252 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:06.527259 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:06.527266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:06.527288 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:06.527326 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:06.529515 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:06.530278 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:06.530284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:06.530291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:06.530431 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:06.530796 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:06.530800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:06.530804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:06.531390 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:06.531601 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:06.531932 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 20:19:06.532193 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260924 20:19:06.533361 18616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:06.535486 18622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } } }
I20260924 20:19:06.535727 18622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:06.535948 18623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e67f2970a544cbc88c3c9cfa730db0e. Latest consensus state: current_term: 1 leader_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e67f2970a544cbc88c3c9cfa730db0e" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 41401 } } }
I20260924 20:19:06.536051 18623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:06.537158 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 20:19:06.538314 18635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:06.538437 18635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:06.538583 18636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:06.539477 18636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:06.545970 18636 catalog_manager.cc:1347] Generated new cluster ID: 6b94f33b740f4a3984d3ce63fe291fee
I20260924 20:19:06.546175 18636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:06.551486 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:06.539660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56761 (local address 127.29.83.190:41401)
0924 20:19:06.539785 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:06.539791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:06.540329 (+ 538us) server_negotiation.cc:407] Connection header received
0924 20:19:06.540540 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:06.540543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:06.540597 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:06.540652 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:06.541675 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:06.542621 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:06.544327 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:06.544727 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:06.544834 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:06.546903 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:06.546908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:06.546911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:06.546941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:06.549058 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:06.549838 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:06.549842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:06.549844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:06.549942 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:06.550351 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:06.550354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:06.550356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:06.550622 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:06.550759 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:06.551203 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 20:19:06.551310 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:19:06.566929 18636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:06.568648 18636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:06.594264 18636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e67f2970a544cbc88c3c9cfa730db0e: Generated new TSK 0
I20260924 20:19:06.595114 18636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:06.603430 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18548
W20260924 20:19:06.612310 18533 connection.cc:535] client connection to 127.29.83.190:41401 recv error: Network error: failed to read from TLS socket (remote: 127.29.83.190:41401): Connection reset by peer (error 104)
2026-09-24T20:19:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18647](info): set up 2 sockets
Sep 24 20:19:06 dist-test-slave-52fr krb5kdc[18647](info): commencing operation
krb5kdc: starting...
W20260924 20:19:07.658177 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:07 dist-test-slave-52fr krb5kdc[18647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:07.784811 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40069
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:07.901298 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:07.901661 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:07.901742 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:07.906163 18660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:07.906251 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:07.906270 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:07.906288 18660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:07.906304 18660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:07.912268 18660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:07.913403 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:07.914598 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:07.923769 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:07.923830 18660 server_base.cc:1034] running on GCE node
W20260924 20:19:07.923779 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:07.923769 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:07.924703 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:07.926005 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:07.927184 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790281147927139 us; error 60 us; skew 500 ppm
Sep 24 20:19:07 dist-test-slave-52fr krb5kdc[18647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281147, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:07.936347 18660 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:07.937395 18660 webserver.cc:460] Webserver started at http://127.29.83.190:37157/ using document root <none> and password file <none>
I20260924 20:19:07.938114 18660 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:07.938220 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:07.938488 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:07.940928 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "81afea66d72d442884c3016f39b37d8d"
format_stamp: "Formatted at 2026-09-24 20:19:07 on dist-test-slave-52fr"
server_key: "f820a2a18de22b35009910c744cfb3b7"
server_key_iv: "6738ed4ef4a9803ead46d09b2c2b6c99"
server_key_version: "encryptionkey@0"
I20260924 20:19:07.941573 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "81afea66d72d442884c3016f39b37d8d"
format_stamp: "Formatted at 2026-09-24 20:19:07 on dist-test-slave-52fr"
server_key: "f820a2a18de22b35009910c744cfb3b7"
server_key_iv: "6738ed4ef4a9803ead46d09b2c2b6c99"
server_key_version: "encryptionkey@0"
I20260924 20:19:07.946003 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:19:07.949038 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:07.950595 18660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:07.950771 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "81afea66d72d442884c3016f39b37d8d"
format_stamp: "Formatted at 2026-09-24 20:19:07 on dist-test-slave-52fr"
server_key: "f820a2a18de22b35009910c744cfb3b7"
server_key_iv: "6738ed4ef4a9803ead46d09b2c2b6c99"
server_key_version: "encryptionkey@0"
I20260924 20:19:07.950914 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:07.966984 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:07.970970 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:07.971216 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:07.979276 18660 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40069
I20260924 20:19:07.979300 18727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40069 every 8 connection(s)
I20260924 20:19:07.980295 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:07.981801 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18660
I20260924 20:19:07.981957 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:07.982267 30030 external_mini_cluster.cc:1428] Setting key d20a888ba7c8011f2ab33aed6ee5999d
I20260924 20:19:07.983609 18728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:07.989398 18728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: Bootstrap starting.
I20260924 20:19:07.991586 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:07.983723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44561 (local address 127.29.83.190:40069)
0924 20:19:07.984185 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:07.984200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:07.984236 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:19:07.984879 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:07.984895 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:07.985229 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:07.985592 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:07.986037 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:07.987048 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:07.988002 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:07.988511 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:07.988683 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:07.989720 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:07.989725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:07.989738 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:07.990220 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:07.990278 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:07.990303 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:07.990474 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:07.990659 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 20:19:07.990816 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260924 20:19:07.991968 18728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:07.992719 18728 log.cc:824] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:07.994951 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:07.995177 18728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: No bootstrap required, opened a new log
I20260924 20:19:07.999588 18728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } }
I20260924 20:19:07.999899 18728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:08.000020 18728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81afea66d72d442884c3016f39b37d8d, State: Initialized, Role: FOLLOWER
I20260924 20:19:08.000562 18728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } }
I20260924 20:19:08.000739 18728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:08.000820 18728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:08.000934 18728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:08.001456 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:07.997063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49263 (local address 127.29.83.190:40069)
0924 20:19:07.997180 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:07.997184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:07.997489 (+ 305us) server_negotiation.cc:407] Connection header received
0924 20:19:07.997631 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:07.997633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:07.997672 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:07.997740 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:07.998064 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:07.998566 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:07.999661 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:07.999886 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:08.000005 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:08.000912 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:08.000916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:08.000919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:08.000933 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:08.000959 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:08.000978 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:08.001091 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:08.001245 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 20:19:08.001284 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:19:08.002238 18728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } }
I20260924 20:19:08.002619 18728 leader_election.cc:302] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81afea66d72d442884c3016f39b37d8d; no voters:
I20260924 20:19:08.002899 18728 leader_election.cc:288] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:08.003026 18738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:08.003279 18738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [term 1 LEADER]: Becoming Leader. State: Replica: 81afea66d72d442884c3016f39b37d8d, State: Running, Role: LEADER
I20260924 20:19:08.003649 18738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } }
I20260924 20:19:08.004235 18728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:08.005441 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81afea66d72d442884c3016f39b37d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } } }
I20260924 20:19:08.005466 18740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81afea66d72d442884c3016f39b37d8d. Latest consensus state: current_term: 1 leader_uuid: "81afea66d72d442884c3016f39b37d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afea66d72d442884c3016f39b37d8d" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40069 } } }
I20260924 20:19:08.005647 18740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:08.005632 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d [sys.catalog]: This master's current role is: LEADER
W20260924 20:19:08.009197 18753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:08.009265 18753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:08.009351 18749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:08.010139 18749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:08.015527 18749 catalog_manager.cc:1347] Generated new cluster ID: 1fccdb16a0f54bc4a0b6f557dd7c584f
I20260924 20:19:08.015605 18749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:08.033138 18749 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:08.034549 18749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:08.041795 18749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81afea66d72d442884c3016f39b37d8d: Generated new TSK 0
I20260924 20:19:08.042711 18749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:08.050372 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18660
W20260924 20:19:08.055506 18645 connection.cc:535] client connection to 127.29.83.190:40069 recv error: Network error: failed to read from TLS socket (remote: 127.29.83.190:40069): Connection reset by peer (error 104)
2026-09-24T20:19:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:08 dist-test-slave-52fr krb5kdc[18759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:08 dist-test-slave-52fr krb5kdc[18759](info): set up 2 sockets
Sep 24 20:19:08 dist-test-slave-52fr krb5kdc[18759](info): commencing operation
krb5kdc: starting...
W20260924 20:19:09.098181 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:09 dist-test-slave-52fr krb5kdc[18759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:09.222731 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46811
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:46811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:09.334224 18772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:09.334686 18772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:09.334786 18772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:09.341307 18772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:09.341398 18772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:09.341445 18772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:09.341511 18772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:09.341544 18772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:09.347466 18772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:46811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:46811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:09.348654 18772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:09.349692 18772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:09.358156 18777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:09.358127 18780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:09.358150 18778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:09.358386 18772 server_base.cc:1034] running on GCE node
I20260924 20:19:09.359241 18772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:09.360570 18772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:09.361811 18772 hybrid_clock.cc:648] HybridClock initialized: now 1790281149361731 us; error 37 us; skew 500 ppm
Sep 24 20:19:09 dist-test-slave-52fr krb5kdc[18759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281149, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:09.364668 18772 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:09.365548 18772 webserver.cc:460] Webserver started at http://127.29.83.190:43549/ using document root <none> and password file <none>
I20260924 20:19:09.366159 18772 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:09.366223 18772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:09.366400 18772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:09.368474 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "5a20f01ea33a4a78ac56181f663e6d17"
format_stamp: "Formatted at 2026-09-24 20:19:09 on dist-test-slave-52fr"
server_key: "68b38000fa1a8816298dda8d0d4e22ce"
server_key_iv: "c606ea082ee0d74fb896b1cc1dbc3173"
server_key_version: "encryptionkey@0"
I20260924 20:19:09.369014 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "5a20f01ea33a4a78ac56181f663e6d17"
format_stamp: "Formatted at 2026-09-24 20:19:09 on dist-test-slave-52fr"
server_key: "68b38000fa1a8816298dda8d0d4e22ce"
server_key_iv: "c606ea082ee0d74fb896b1cc1dbc3173"
server_key_version: "encryptionkey@0"
I20260924 20:19:09.372989 18772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:09.375588 18787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:09.376922 18772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:19:09.377084 18772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "5a20f01ea33a4a78ac56181f663e6d17"
format_stamp: "Formatted at 2026-09-24 20:19:09 on dist-test-slave-52fr"
server_key: "68b38000fa1a8816298dda8d0d4e22ce"
server_key_iv: "c606ea082ee0d74fb896b1cc1dbc3173"
server_key_version: "encryptionkey@0"
I20260924 20:19:09.377215 18772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:09.382066 18772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:09.385367 18772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:09.385584 18772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:09.392757 18772 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:46811
I20260924 20:19:09.392789 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:46811 every 8 connection(s)
I20260924 20:19:09.393687 18772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:09.396423 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:09.399194 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18772
I20260924 20:19:09.399336 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:09.399608 30030 external_mini_cluster.cc:1428] Setting key 4299aa2ad030a23c03a7f0a7276408e4
I20260924 20:19:09.402396 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17: Bootstrap starting.
I20260924 20:19:09.404702 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:09.400971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50787 (local address 127.29.83.190:46811)
0924 20:19:09.401408 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:09.401423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:09.401456 (+ 33us) server_negotiation.cc:407] Connection header received
0924 20:19:09.402134 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:09.402154 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:09.402443 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:09.402802 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:09.403578 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:09.403582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:09.403594 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:09.403604 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:09.403654 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:09.403672 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:09.403818 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:09.403890 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 20:19:09.404049 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260924 20:19:09.404769 18840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:09.405498 18840 log.cc:824] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:09.407480 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:09.407707 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17: No bootstrap required, opened a new log
I20260924 20:19:09.411268 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:09.409469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38209 (local address 127.29.83.190:46811)
0924 20:19:09.409593 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:09.409598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:09.409810 (+ 212us) server_negotiation.cc:407] Connection header received
0924 20:19:09.409909 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:09.409911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:09.409937 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:09.410000 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:09.410750 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:09.410753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:09.410755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:09.410784 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:09.410808 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:09.410817 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:09.410895 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:09.411062 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 20:19:09.411098 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:19:09.412024 18840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } }
I20260924 20:19:09.412338 18840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:09.412402 18840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a20f01ea33a4a78ac56181f663e6d17, State: Initialized, Role: FOLLOWER
I20260924 20:19:09.412849 18840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } }
I20260924 20:19:09.413010 18840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:09.413100 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:09.413206 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:09.414206 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } }
I20260924 20:19:09.414551 18840 leader_election.cc:302] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a20f01ea33a4a78ac56181f663e6d17; no voters:
I20260924 20:19:09.414876 18840 leader_election.cc:288] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:09.415053 18850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:09.415383 18850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [term 1 LEADER]: Becoming Leader. State: Replica: 5a20f01ea33a4a78ac56181f663e6d17, State: Running, Role: LEADER
I20260924 20:19:09.415789 18850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } }
I20260924 20:19:09.416142 18840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:09.417529 18851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a20f01ea33a4a78ac56181f663e6d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } } }
I20260924 20:19:09.417502 18852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a20f01ea33a4a78ac56181f663e6d17. Latest consensus state: current_term: 1 leader_uuid: "5a20f01ea33a4a78ac56181f663e6d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a20f01ea33a4a78ac56181f663e6d17" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 46811 } } }
I20260924 20:19:09.417642 18851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:09.417642 18852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:09.418119 18860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:09.421366 18860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:09.427126 18860 catalog_manager.cc:1347] Generated new cluster ID: 4b8d335c3e0743f18adadf60fdabcf24
I20260924 20:19:09.427214 18860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:09.437104 18860 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:09.437449 18860 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:09.446038 18860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a20f01ea33a4a78ac56181f663e6d17: Generated new TSK 0
I20260924 20:19:09.446761 18860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:09.449626 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18772
2026-09-24T20:19:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:09 dist-test-slave-52fr krb5kdc[18871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:09 dist-test-slave-52fr krb5kdc[18871](info): set up 2 sockets
Sep 24 20:19:09 dist-test-slave-52fr krb5kdc[18871](info): commencing operation
krb5kdc: starting...
W20260924 20:19:10.498345 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:10.624833 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40103
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:10.743014 18884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:10.743359 18884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:10.743439 18884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:10.747764 18884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:10.747898 18884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:10.747941 18884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:10.747982 18884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:10.748016 18884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:10.753849 18884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:10.754979 18884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:10.756075 18884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:10.765159 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:10.765298 18884 server_base.cc:1034] running on GCE node
W20260924 20:19:10.765229 18892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:10.765235 18890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:10.766100 18884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:10.767503 18884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:10.768711 18884 hybrid_clock.cc:648] HybridClock initialized: now 1790281150768671 us; error 60 us; skew 500 ppm
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281150, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:10.771926 18884 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:10.772892 18884 webserver.cc:460] Webserver started at http://127.29.83.190:40801/ using document root <none> and password file <none>
I20260924 20:19:10.773458 18884 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:10.773525 18884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:10.773696 18884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:10.775869 18884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "b28335ffe6714a21802da1b03103f547"
format_stamp: "Formatted at 2026-09-24 20:19:10 on dist-test-slave-52fr"
server_key: "89072c5b2e58600294326e6d083d2f5d"
server_key_iv: "d6a6a253aa5022239eddb8587c24b6fe"
server_key_version: "encryptionkey@0"
I20260924 20:19:10.776415 18884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "b28335ffe6714a21802da1b03103f547"
format_stamp: "Formatted at 2026-09-24 20:19:10 on dist-test-slave-52fr"
server_key: "89072c5b2e58600294326e6d083d2f5d"
server_key_iv: "d6a6a253aa5022239eddb8587c24b6fe"
server_key_version: "encryptionkey@0"
I20260924 20:19:10.780238 18884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:19:10.782882 18899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:10.784174 18884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:19:10.784366 18884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "b28335ffe6714a21802da1b03103f547"
format_stamp: "Formatted at 2026-09-24 20:19:10 on dist-test-slave-52fr"
server_key: "89072c5b2e58600294326e6d083d2f5d"
server_key_iv: "d6a6a253aa5022239eddb8587c24b6fe"
server_key_version: "encryptionkey@0"
I20260924 20:19:10.784477 18884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:10.802248 18884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:10.805722 18884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:10.806043 18884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:10.813848 18884 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40103
I20260924 20:19:10.813892 18951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40103 every 8 connection(s)
I20260924 20:19:10.814901 18884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:10.817855 18952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:10.821574 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18884
I20260924 20:19:10.821699 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:10.822042 30030 external_mini_cluster.cc:1428] Setting key a32d067104724a28be18444722170577
I20260924 20:19:10.824219 18952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547: Bootstrap starting.
I20260924 20:19:10.826581 18952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:10.827365 18952 log.cc:824] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:10.829321 18952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547: No bootstrap required, opened a new log
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:10.833269 18952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } }
I20260924 20:19:10.833537 18952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:10.833630 18952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28335ffe6714a21802da1b03103f547, State: Initialized, Role: FOLLOWER
I20260924 20:19:10.834223 18952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } }
I20260924 20:19:10.834405 18952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:10.834514 18952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:10.834636 18952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:10.835901 18952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } }
I20260924 20:19:10.836270 18952 leader_election.cc:302] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b28335ffe6714a21802da1b03103f547; no voters:
I20260924 20:19:10.836624 18952 leader_election.cc:288] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:10.836764 18957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:10.837018 18957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [term 1 LEADER]: Becoming Leader. State: Replica: b28335ffe6714a21802da1b03103f547, State: Running, Role: LEADER
I20260924 20:19:10.837142 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:10.823515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43965 (local address 127.29.83.190:40103)
0924 20:19:10.824012 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:10.824027 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:10.824065 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:19:10.824825 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:10.824845 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:10.825215 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:10.825573 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:10.826450 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:10.827584 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:10.828503 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:10.829012 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:10.829197 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:10.831819 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:10.831825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:10.831840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:10.831869 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:10.833686 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:10.834223 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:10.834228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:10.834233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:10.834361 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:10.834708 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:10.834712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:10.834714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:10.835239 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:10.835459 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:10.835769 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 20:19:10.836096 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":150,"threads_started":1}
I20260924 20:19:10.837396 18957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } }
I20260924 20:19:10.837966 18952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:10.839931 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b28335ffe6714a21802da1b03103f547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } } }
I20260924 20:19:10.840055 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:10.840613 18972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:10.841063 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:10.840297 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b28335ffe6714a21802da1b03103f547. Latest consensus state: current_term: 1 leader_uuid: "b28335ffe6714a21802da1b03103f547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28335ffe6714a21802da1b03103f547" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40103 } } }
I20260924 20:19:10.841876 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:10.844030 18972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:10.849368 18972 catalog_manager.cc:1347] Generated new cluster ID: 69ba81eed12345338206a205aacc42d8
I20260924 20:19:10.849440 18972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:10.855782 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:10.843237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37401 (local address 127.29.83.190:40103)
0924 20:19:10.843407 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:10.843411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:10.847533 (+ 4122us) server_negotiation.cc:407] Connection header received
0924 20:19:10.847699 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:10.847705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:10.847757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:10.847854 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:10.848741 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:10.849529 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:10.850460 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:10.850714 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:10.850845 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:10.852368 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:10.852372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:10.852374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:10.852402 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:10.853951 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:10.854394 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:10.854398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:10.854400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:10.854483 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:10.854765 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:10.854768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:10.854770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:10.854958 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:10.855089 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:10.855487 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 20:19:10.855616 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 20:19:10.862896 18972 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:10.863881 18972 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:10.870270 18972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b28335ffe6714a21802da1b03103f547: Generated new TSK 0
I20260924 20:19:10.871006 18972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:10.876026 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18884
2026-09-24T20:19:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18983](info): set up 2 sockets
Sep 24 20:19:10 dist-test-slave-52fr krb5kdc[18983](info): commencing operation
krb5kdc: starting...
W20260924 20:19:11.918313 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:11 dist-test-slave-52fr krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:12.041326 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34625
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:34625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:12.155797 18996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:12.156083 18996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:12.156126 18996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:12.160252 18996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:12.160327 18996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:12.160346 18996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:12.160363 18996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:12.160379 18996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:12.165671 18996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:34625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:12.166694 18996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:12.167666 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:12.176165 19004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:12.176263 18996 server_base.cc:1034] running on GCE node
W20260924 20:19:12.176152 19001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:12.176155 19002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:12.177121 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:12.178367 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:12.179605 18996 hybrid_clock.cc:648] HybridClock initialized: now 1790281152179576 us; error 42 us; skew 500 ppm
Sep 24 20:19:12 dist-test-slave-52fr krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281152, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:12.182545 18996 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:12.183470 18996 webserver.cc:460] Webserver started at http://127.29.83.190:35539/ using document root <none> and password file <none>
I20260924 20:19:12.184057 18996 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:12.184154 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:12.184401 18996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:12.186560 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "99b2da10d47c4f4ea3a1da220003ec5f"
format_stamp: "Formatted at 2026-09-24 20:19:12 on dist-test-slave-52fr"
server_key: "ef2f2e3838e4d27101c3409f55c17e06"
server_key_iv: "1359fd2562561374252e386adcead792"
server_key_version: "encryptionkey@0"
I20260924 20:19:12.187125 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "99b2da10d47c4f4ea3a1da220003ec5f"
format_stamp: "Formatted at 2026-09-24 20:19:12 on dist-test-slave-52fr"
server_key: "ef2f2e3838e4d27101c3409f55c17e06"
server_key_iv: "1359fd2562561374252e386adcead792"
server_key_version: "encryptionkey@0"
I20260924 20:19:12.190954 18996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 20:19:12.193424 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:12.194718 18996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:19:12.194869 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "99b2da10d47c4f4ea3a1da220003ec5f"
format_stamp: "Formatted at 2026-09-24 20:19:12 on dist-test-slave-52fr"
server_key: "ef2f2e3838e4d27101c3409f55c17e06"
server_key_iv: "1359fd2562561374252e386adcead792"
server_key_version: "encryptionkey@0"
I20260924 20:19:12.194984 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:12.206321 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:12.210450 18996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:12.210692 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:12.218389 18996 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:34625
I20260924 20:19:12.218462 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:34625 every 8 connection(s)
I20260924 20:19:12.219506 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:12.222651 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:12.228341 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 18996
I20260924 20:19:12.228502 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:12.228803 30030 external_mini_cluster.cc:1428] Setting key c505041212cef85b2be96ab57feb542c
I20260924 20:19:12.228772 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f: Bootstrap starting.
I20260924 20:19:12.231559 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:12.232380 19064 log.cc:824] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:12.234875 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f: No bootstrap required, opened a new log
I20260924 20:19:12.239084 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:12.230321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.29.83.190:34625)
0924 20:19:12.230810 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:12.230828 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:12.230867 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:19:12.231826 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:12.231848 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:12.232228 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:12.232596 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:12.233042 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:12.234228 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:12.235518 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:12.236012 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:12.236137 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:12.237125 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:12.237130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:12.237145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:12.237596 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:12.237654 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:12.237677 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:12.237935 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:12.238000 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 20:19:12.238250 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":177,"threads_started":1}
I20260924 20:19:12.239326 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } }
I20260924 20:19:12.239605 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:12.239696 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b2da10d47c4f4ea3a1da220003ec5f, State: Initialized, Role: FOLLOWER
I20260924 20:19:12.240180 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } }
I20260924 20:19:12.240350 19064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:12.240417 19064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:12.240514 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:12.241724 19064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } }
I20260924 20:19:12.242194 19064 leader_election.cc:302] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99b2da10d47c4f4ea3a1da220003ec5f; no voters:
I20260924 20:19:12.242455 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:12.242594 19064 leader_election.cc:288] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:12.242722 19069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:12.243016 19069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [term 1 LEADER]: Becoming Leader. State: Replica: 99b2da10d47c4f4ea3a1da220003ec5f, State: Running, Role: LEADER
I20260924 20:19:12.243405 19069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } }
I20260924 20:19:12.244043 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:12.245154 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } } }
I20260924 20:19:12.245309 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:12.245795 19083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:12.246223 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b2da10d47c4f4ea3a1da220003ec5f. Latest consensus state: current_term: 1 leader_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b2da10d47c4f4ea3a1da220003ec5f" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 34625 } } }
I20260924 20:19:12.246392 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:12.249840 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:12.245576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52643 (local address 127.29.83.190:34625)
0924 20:19:12.245750 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:12.245780 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:12.245797 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:19:12.245844 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:12.245849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:12.245909 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:12.246006 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:12.246327 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:12.247075 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:12.247937 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:12.248277 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:12.248380 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:12.249298 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:12.249302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:12.249305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:12.249318 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:12.249350 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:12.249364 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:12.249437 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:12.249604 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 20:19:12.249670 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 20:19:12.250315 19083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:12.256364 19083 catalog_manager.cc:1347] Generated new cluster ID: 33c9c4283c424ecd85951ecced7fc3b9
I20260924 20:19:12.256448 19083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:12.265568 19083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:12.266743 19083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:12.271056 19083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99b2da10d47c4f4ea3a1da220003ec5f: Generated new TSK 0
I20260924 20:19:12.271705 19083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:12.279212 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 18996
2026-09-24T20:19:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:12 dist-test-slave-52fr krb5kdc[19095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:12 dist-test-slave-52fr krb5kdc[19095](info): set up 2 sockets
Sep 24 20:19:12 dist-test-slave-52fr krb5kdc[19095](info): commencing operation
krb5kdc: starting...
W20260924 20:19:13.330114 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:13 dist-test-slave-52fr krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:13.458518 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36865
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:13.578586 19108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:13.578924 19108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:13.578971 19108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:13.583271 19108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:13.583353 19108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:13.583374 19108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:13.583392 19108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:13.583408 19108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:13.589155 19108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:13.590318 19108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:13.591370 19108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:13.601290 19116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:13.601320 19113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:13.601683 19108 server_base.cc:1034] running on GCE node
W20260924 20:19:13.601330 19114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:13.602669 19108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:13.603899 19108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:13.605108 19108 hybrid_clock.cc:648] HybridClock initialized: now 1790281153605048 us; error 66 us; skew 500 ppm
Sep 24 20:19:13 dist-test-slave-52fr krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281153, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:13.612376 19108 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:13.613483 19108 webserver.cc:460] Webserver started at http://127.29.83.190:40215/ using document root <none> and password file <none>
I20260924 20:19:13.614204 19108 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:13.614305 19108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:13.614555 19108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:13.616719 19108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "69f340f4e91e4d989cfa94bebf7942d2"
format_stamp: "Formatted at 2026-09-24 20:19:13 on dist-test-slave-52fr"
server_key: "507388a43103518d40d31e8a8766f352"
server_key_iv: "7a0edf9c0d6c31e89d5b1a1b7b612169"
server_key_version: "encryptionkey@0"
I20260924 20:19:13.617331 19108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "69f340f4e91e4d989cfa94bebf7942d2"
format_stamp: "Formatted at 2026-09-24 20:19:13 on dist-test-slave-52fr"
server_key: "507388a43103518d40d31e8a8766f352"
server_key_iv: "7a0edf9c0d6c31e89d5b1a1b7b612169"
server_key_version: "encryptionkey@0"
I20260924 20:19:13.621546 19108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:13.624509 19123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:13.626001 19108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:13.626157 19108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "69f340f4e91e4d989cfa94bebf7942d2"
format_stamp: "Formatted at 2026-09-24 20:19:13 on dist-test-slave-52fr"
server_key: "507388a43103518d40d31e8a8766f352"
server_key_iv: "7a0edf9c0d6c31e89d5b1a1b7b612169"
server_key_version: "encryptionkey@0"
I20260924 20:19:13.626291 19108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:13.631245 19108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:13.634667 19108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:13.634881 19108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:13.642642 19108 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:36865
I20260924 20:19:13.642711 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:36865 every 8 connection(s)
I20260924 20:19:13.643664 19108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:13.646361 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19108
I20260924 20:19:13.646520 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:13.646737 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:13.646858 30030 external_mini_cluster.cc:1428] Setting key 7a59a28e1b297ba76af934a0ad4cd978
I20260924 20:19:13.652027 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:13.648226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52005 (local address 127.29.83.190:36865)
0924 20:19:13.648632 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:13.648647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:13.648682 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:19:13.649325 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:13.649341 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:13.649621 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:13.650021 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:13.650840 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:13.650844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:13.650856 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:13.650866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:13.650918 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:13.650938 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:13.651085 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:13.651157 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 20:19:13.651329 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260924 20:19:13.654464 19176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2: Bootstrap starting.
I20260924 20:19:13.655406 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:13.657272 19176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:13.658341 19176 log.cc:824] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:13.659896 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:13.657551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40331 (local address 127.29.83.190:36865)
0924 20:19:13.657821 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:13.657826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:13.658082 (+ 256us) server_negotiation.cc:407] Connection header received
0924 20:19:13.658265 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:13.658268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:13.658305 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:13.658380 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:13.659372 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:13.659377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:13.659380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:19:13.659392 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:13.659423 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:19:13.659440 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:13.659544 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:13.659681 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 20:19:13.659738 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 20:19:13.661252 19176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2: No bootstrap required, opened a new log
I20260924 20:19:13.665398 19176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } }
I20260924 20:19:13.665669 19176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:13.665705 19176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f340f4e91e4d989cfa94bebf7942d2, State: Initialized, Role: FOLLOWER
I20260924 20:19:13.666226 19176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } }
I20260924 20:19:13.666391 19176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:13.666463 19176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:13.666569 19176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:13.667761 19176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } }
I20260924 20:19:13.668123 19176 leader_election.cc:302] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f340f4e91e4d989cfa94bebf7942d2; no voters:
I20260924 20:19:13.668447 19176 leader_election.cc:288] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:13.668751 19186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:13.669032 19186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [term 1 LEADER]: Becoming Leader. State: Replica: 69f340f4e91e4d989cfa94bebf7942d2, State: Running, Role: LEADER
I20260924 20:19:13.669423 19186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } }
I20260924 20:19:13.669952 19176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:13.671350 19188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f340f4e91e4d989cfa94bebf7942d2. Latest consensus state: current_term: 1 leader_uuid: "69f340f4e91e4d989cfa94bebf7942d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } } }
I20260924 20:19:13.671350 19187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f340f4e91e4d989cfa94bebf7942d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f340f4e91e4d989cfa94bebf7942d2" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 36865 } } }
I20260924 20:19:13.671525 19188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:13.671574 19187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:13.672050 19197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:13.675192 19197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:13.681265 19197 catalog_manager.cc:1347] Generated new cluster ID: 346bde8388094ad0813f713f4acf5a87
I20260924 20:19:13.681361 19197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:13.691908 19197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:13.692337 19197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:13.699632 19197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f340f4e91e4d989cfa94bebf7942d2: Generated new TSK 0
I20260924 20:19:13.700667 19197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:19:13.715766 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19108
2026-09-24T20:19:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1437 ms)
[----------] 18 tests from AuthTokenIssuingTest (25717 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:13 dist-test-slave-52fr krb5kdc[19207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:13 dist-test-slave-52fr krb5kdc[19207](info): set up 2 sockets
Sep 24 20:19:13 dist-test-slave-52fr krb5kdc[19207](info): commencing operation
krb5kdc: starting...
W20260924 20:19:14.758167 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:14 dist-test-slave-52fr krb5kdc[19207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:14.887228 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40101
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:15.003829 19220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:15.004202 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:15.004282 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:15.004395 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:15.008679 19220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:15.008785 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:15.008832 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:15.008893 19220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:15.008934 19220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:15.014540 19220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:15.015751 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:15.016913 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:15.026475 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.026475 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.026621 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:15.027724 19220 server_base.cc:1034] running on GCE node
I20260924 20:19:15.028389 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:15.029572 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:15.030791 19220 hybrid_clock.cc:648] HybridClock initialized: now 1790281155030755 us; error 49 us; skew 500 ppm
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:15.033977 19220 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:15.035028 19220 webserver.cc:460] Webserver started at http://127.29.83.190:45925/ using document root <none> and password file <none>
I20260924 20:19:15.035609 19220 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:15.035686 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:15.035857 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:15.038130 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "79b3262041454f66b871128a8064f340"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "a375076f91bdf95c8619e1dbcda35247"
server_key_iv: "81a7aa992dfadf34c2a747186f085d3d"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.038709 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "79b3262041454f66b871128a8064f340"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "a375076f91bdf95c8619e1dbcda35247"
server_key_iv: "81a7aa992dfadf34c2a747186f085d3d"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.042858 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:19:15.045644 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:15.047132 19220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:19:15.047290 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "79b3262041454f66b871128a8064f340"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "a375076f91bdf95c8619e1dbcda35247"
server_key_iv: "81a7aa992dfadf34c2a747186f085d3d"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.047426 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:15.082254 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:15.086164 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:15.086400 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:15.094311 19220 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:40101
I20260924 20:19:15.094683 19287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:40101 every 8 connection(s)
I20260924 20:19:15.095446 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:15.098507 19288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:15.102658 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.105113 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19220
I20260924 20:19:15.105300 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:15.105634 30030 external_mini_cluster.cc:1428] Setting key 895f2d45bb97d376ac33cbf1e789786d
WARNING: no policy specified for kudu/127.29.83.189@KRBTEST.COM; defaulting to no policy
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.189@KRBTEST.COM" created.
I20260924 20:19:15.120566 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.105310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.29.83.190:40101)
0924 20:19:15.105879 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.105900 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.106212 (+ 312us) server_negotiation.cc:407] Connection header received
0924 20:19:15.107487 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.107493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.107707 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.107899 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.109152 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.110242 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.111916 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.112294 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.112413 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.115217 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.115223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.115235 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.115258 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.116977 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.117577 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.117584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.117586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.117730 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.118149 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.118156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.118165 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.118807 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.118953 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.119363 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.119740 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":367,"thread_start_us":180,"threads_started":1}
I20260924 20:19:15.120570 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.105310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40101 (local address 127.0.0.1:55104)
0924 20:19:15.106044 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.106088 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.107037 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.107990 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.107999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.108540 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.108961 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.108984 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.110381 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.110385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.111777 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.111794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.112470 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.112476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.112618 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.113256 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.113278 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.115011 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.117112 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.117124 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.117138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.117432 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.117889 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.117895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.117900 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.118042 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.118938 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.118944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.119080 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.119272 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.119847 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":135,"spinlock_wait_cycles":34048,"thread_start_us":182,"threads_started":1}
I20260924 20:19:15.123205 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.124043 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.123701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:49854)
0924 20:19:15.123858 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.123955 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:19:15.124279 19237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
I20260924 20:19:15.125159 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.124903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:49860)
0924 20:19:15.125034 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.125090 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 20:19:15.125346 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:40749: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
I20260924 20:19:15.135123 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:40749
--webserver_interface=127.29.83.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:19:15.174640 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } attempt: 1
I20260924 20:19:15.175616 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.175246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:49864)
0924 20:19:15.175435 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.175517 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 20:19:15.176548 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.176290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:49878)
0924 20:19:15.176419 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.176480 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:19:15.176769 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:40749: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
I20260924 20:19:15.245003 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } attempt: 2
I20260924 20:19:15.246014 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.245622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:41882)
0924 20:19:15.245826 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.245912 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 20:19:15.246851 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.246604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:41884)
0924 20:19:15.246723 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.246780 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:19:15.247048 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:40749: Network error: Client connection negotiation failed: client connection to 127.29.83.189:40749: connect: Connection refused (error 111)
W20260924 20:19:15.252822 19295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:15.253140 19295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:15.253165 19295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:15.253242 19295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:15.257558 19295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:15.257633 19295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:15.257654 19295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:15.257673 19295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:15.257688 19295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:15.263554 19295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:40749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.83.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:15.264698 19295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:15.265869 19295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:15.275665 19300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.275669 19303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.275665 19301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:15.275749 19295 server_base.cc:1034] running on GCE node
I20260924 20:19:15.276713 19295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:15.278066 19295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:15.279273 19295 hybrid_clock.cc:648] HybridClock initialized: now 1790281155279255 us; error 34 us; skew 500 ppm
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:15.282305 19295 init.cc:377] Logged in from keytab as kudu/127.29.83.189@KRBTEST.COM (short username kudu)
I20260924 20:19:15.283244 19295 webserver.cc:460] Webserver started at http://127.29.83.189:35479/ using document root <none> and password file <none>
I20260924 20:19:15.283847 19295 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:15.283944 19295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:15.284178 19295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:15.286371 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data/instance:
uuid: "ebff7a1acdd6441e964a1e36003c0730"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "ee6727e145ada362fe54dd356db90b05"
server_key_iv: "9aa2931762c3e01cd2f4d556860610c2"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.287014 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal/instance:
uuid: "ebff7a1acdd6441e964a1e36003c0730"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "ee6727e145ada362fe54dd356db90b05"
server_key_iv: "9aa2931762c3e01cd2f4d556860610c2"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.291452 19295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:19:15.294286 19310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:15.295661 19295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:15.295827 19295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "ebff7a1acdd6441e964a1e36003c0730"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "ee6727e145ada362fe54dd356db90b05"
server_key_iv: "9aa2931762c3e01cd2f4d556860610c2"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.295959 19295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:15.311549 19295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:15.315524 19295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:15.315786 19295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:15.323439 19295 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.189:40749
I20260924 20:19:15.323760 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.189:40749 every 8 connection(s)
I20260924 20:19:15.324520 19295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
I20260924 20:19:15.327561 19363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:15.331719 19363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.332494 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19295
I20260924 20:19:15.332645 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-1/wal/instance
I20260924 20:19:15.332969 30030 external_mini_cluster.cc:1428] Setting key c44d0dcb6f878948d47ef71f4793212f
I20260924 20:19:15.338275 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } attempt: 3
WARNING: no policy specified for kudu/127.29.83.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.188@KRBTEST.COM" created.
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:19:15.354245 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.340129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:41898)
0924 20:19:15.340507 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.340524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.340621 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.343846 (+ 3225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.343855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.344062 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.344192 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.344200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.345445 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.345453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.346117 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.346127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.346873 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.346876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.347027 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.347763 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.347784 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.349441 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.352221 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.352224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.352226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.352501 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.352918 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.352921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.352922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.353065 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.353699 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.353703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.353828 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.353941 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.354095 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260924 20:19:15.354574 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.334641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40101 (local address 127.0.0.1:52294)
0924 20:19:15.335327 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.335412 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.336764 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.337384 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.337396 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.337939 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.338718 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.338735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.339918 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.339938 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.341994 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.342008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.342967 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.342971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.343216 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.343931 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.343954 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.348307 (+ 4353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.350576 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.350587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.350599 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.351066 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.351398 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.351401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.351403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.351609 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.352125 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.352132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.352592 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.352809 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.353229 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":174,"threads_started":1}
I20260924 20:19:15.354947 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.336605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.29.83.190:40101)
0924 20:19:15.336823 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.336827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.336876 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:19:15.336993 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.336996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.337083 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.337178 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.338871 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.339767 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.342174 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.342670 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.342782 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.348469 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.348473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.348475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.348502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.350474 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.351184 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.351187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.351189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.351277 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.351719 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.351724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.351726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.351978 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.352095 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.354621 (+ 2526us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.354761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:19:15.355305 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.340946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.29.83.189:40749)
0924 20:19:15.342930 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.342944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.342973 (+ 29us) server_negotiation.cc:407] Connection header received
0924 20:19:15.343028 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.343035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.343299 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.343506 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.344328 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.345266 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.346295 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.346692 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.346840 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.349622 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.349631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.349653 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.349686 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.352110 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.352623 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.352650 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.352654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.352797 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.353178 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.353183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.353188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.353545 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.353704 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.354940 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.355094 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1858,"spinlock_wait_cycles":2304,"thread_start_us":906,"threads_started":1}
I20260924 20:19:15.356329 19363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.357384 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.358152 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.357893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36480)
0924 20:19:15.358028 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.358083 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 20:19:15.358914 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.358603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36486)
0924 20:19:15.358788 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.358843 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 20:19:15.359088 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:19:15.365057 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.356870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:41908)
0924 20:19:15.356972 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.356986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.357060 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.357311 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.357314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.357608 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.357794 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.357806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.358776 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.358778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.359465 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.359474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.360102 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.360105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.360196 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.360715 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.360732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.361972 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.363675 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.363679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.363682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.363930 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.364239 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.364241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.364243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.364319 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.364669 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.364674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.364760 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.364846 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.364945 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 20:19:15.365137 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.357019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.29.83.189:40749)
0924 20:19:15.357125 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.357131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.357141 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:19:15.357167 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.357171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.357217 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.357840 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.357948 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.358625 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.359586 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.359982 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.360093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.362089 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.362092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.362094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.362114 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.363531 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.364043 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.364048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.364053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.364137 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.364382 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.364384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.364385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.364539 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.364647 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.364917 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.365010 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
I20260924 20:19:15.366219 19363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.367038 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.366684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36488)
0924 20:19:15.366853 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.366957 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:19:15.367125 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:44581
--webserver_interface=127.29.83.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:15.367285 19313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.368315 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.368042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36490)
0924 20:19:15.368178 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.368240 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:19:15.368531 19363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.390264 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 1
I20260924 20:19:15.391182 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.390838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36498)
0924 20:19:15.391011 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.391085 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 20:19:15.391964 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.391742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36502)
0924 20:19:15.391855 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.391908 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 20:19:15.392117 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.417891 19363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 1
I20260924 20:19:15.418874 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.418526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36518)
0924 20:19:15.418708 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.418781 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 20:19:15.419744 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.419543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36534)
0924 20:19:15.419642 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.419691 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:19:15.419927 19363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.467298 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 2
I20260924 20:19:15.468295 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.467926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36546)
0924 20:19:15.468114 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.468191 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:19:15.469278 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.468992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36562)
0924 20:19:15.469135 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.469194 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 20:19:15.469509 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.488123 19363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 2
W20260924 20:19:15.488243 19370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:15.488720 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:15.488783 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:15.488838 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:19:15.489149 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.488815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36578)
0924 20:19:15.488993 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.489059 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 20:19:15.490247 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.489976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36590)
0924 20:19:15.490113 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.490177 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 20:19:15.490456 19363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
W20260924 20:19:15.493985 19370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:15.494056 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:15.494077 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:15.494095 19370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:15.494112 19370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:15.500335 19370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:40101,127.29.83.189:40749,127.29.83.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.83.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:15.501610 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:15.502756 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:19:15.512990 19370 server_base.cc:1034] running on GCE node
W20260924 20:19:15.512931 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.512923 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:15.512939 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:15.513935 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:15.515116 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:15.516321 19370 hybrid_clock.cc:648] HybridClock initialized: now 1790281155516302 us; error 32 us; skew 500 ppm
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:15.519279 19370 init.cc:377] Logged in from keytab as kudu/127.29.83.188@KRBTEST.COM (short username kudu)
I20260924 20:19:15.520376 19370 webserver.cc:460] Webserver started at http://127.29.83.188:37657/ using document root <none> and password file <none>
I20260924 20:19:15.520977 19370 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:15.521071 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:15.521302 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:15.523543 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data/instance:
uuid: "bcb683d26da34e5bb3de7a2c7bc273ca"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "587ccb0c31f976d62296b8eae1fc126b"
server_key_iv: "600ea9613c7ef0246e6331af74c38faf"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.524159 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal/instance:
uuid: "bcb683d26da34e5bb3de7a2c7bc273ca"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "587ccb0c31f976d62296b8eae1fc126b"
server_key_iv: "600ea9613c7ef0246e6331af74c38faf"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.528223 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:19:15.531087 19385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:15.532510 19370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:15.532681 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "bcb683d26da34e5bb3de7a2c7bc273ca"
format_stamp: "Formatted at 2026-09-24 20:19:15 on dist-test-slave-52fr"
server_key: "587ccb0c31f976d62296b8eae1fc126b"
server_key_iv: "600ea9613c7ef0246e6331af74c38faf"
server_key_version: "encryptionkey@0"
I20260924 20:19:15.532816 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:15.562737 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:15.566782 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:15.567029 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:15.568768 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 3
I20260924 20:19:15.569723 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.569392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36602)
0924 20:19:15.569576 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.569641 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 20:19:15.570748 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.570450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36610)
0924 20:19:15.570606 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.570678 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 20:19:15.570930 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:44581: Network error: Client connection negotiation failed: client connection to 127.29.83.188:44581: connect: Connection refused (error 111)
I20260924 20:19:15.575655 19370 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.188:44581
I20260924 20:19:15.575956 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.188:44581 every 8 connection(s)
I20260924 20:19:15.576838 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
I20260924 20:19:15.580212 19438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:15.581799 19363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 3
I20260924 20:19:15.584904 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19370
I20260924 20:19:15.585044 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280784619584-30030-0/minicluster-data/master-2/wal/instance
I20260924 20:19:15.585332 30030 external_mini_cluster.cc:1428] Setting key 7256e1261bd35cfc08bc92c0cbd63841
I20260924 20:19:15.585673 19438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:15.597550 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.582758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36612)
0924 20:19:15.582960 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.582980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.583098 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.585828 (+ 2730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.585833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.586072 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.586223 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.586239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.588212 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.588216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.589285 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.589302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.590400 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.590404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.590501 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.591047 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.591067 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.592554 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.595331 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.595334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.595335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.595536 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.595947 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.595951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.595953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.596061 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.597022 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.597027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.597159 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.597266 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.597408 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 20:19:15.598980 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.583477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.29.83.188:44581)
0924 20:19:15.583974 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.583991 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.584030 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:19:15.584836 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.584868 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.585390 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.585881 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.586411 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.588045 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.589394 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.590242 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.590429 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.592761 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.592767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.592779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.592814 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.595213 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.595661 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.595672 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.595684 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.595830 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.596207 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.596214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.596216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.596885 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.597036 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.597546 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.597916 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":175,"threads_started":1}
I20260924 20:19:15.600127 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.586499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.29.83.188:44581)
0924 20:19:15.586891 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.586902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.586917 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:19:15.586958 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.586961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.587006 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.587111 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.588086 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.589154 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.590214 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.590637 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.591122 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.594732 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.594747 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.594749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.594774 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.597562 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.598478 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.598481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.598483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.598569 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.598910 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.598913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.598915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.599146 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.599320 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.599673 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.599819 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":7424,"thread_start_us":168,"threads_started":1}
I20260924 20:19:15.601496 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.588243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.29.83.190:40101)
0924 20:19:15.588369 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.588374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.588784 (+ 410us) server_negotiation.cc:407] Connection header received
0924 20:19:15.589080 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.589084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.589144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.589249 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.590205 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.590931 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.592212 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.592591 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.592713 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.595779 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.595785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.595790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.595826 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.598721 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.599514 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.599518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.599520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.599629 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.600025 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.600029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.600031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.600324 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.600407 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.601155 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.601333 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:19:15.602066 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.588099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:40101 (local address 127.0.0.1:52302)
0924 20:19:15.588640 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.588698 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.588895 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.589309 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.589316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.589884 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.590049 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.590058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.591068 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.591072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.592074 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.592085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.592866 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.592870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.593047 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.593653 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.593669 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.595607 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.598955 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.598965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.598971 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.599375 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.599746 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.599749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.599751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.599854 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.600497 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.600505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.600643 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.601472 (+ 829us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.601703 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":39,"spinlock_wait_cycles":3328,"thread_start_us":107,"threads_started":1}
I20260924 20:19:15.603626 19438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.604544 19363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: Bootstrap starting.
I20260924 20:19:15.607578 19363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: Neither blocks nor log segments found. Creating new log.
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:15.608624 19363 log.cc:824] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:15.612075 19363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: No bootstrap required, opened a new log
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:19:15.613173 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.602128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.29.83.190:40101)
0924 20:19:15.602263 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.602267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.602282 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:19:15.602332 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.602335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.602386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.602474 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.603438 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.604148 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.605123 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.605391 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.605516 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.608735 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.608741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.608743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.608805 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.611008 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.611665 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.611672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.611674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.611798 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.612164 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.612168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.612169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.612421 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.612685 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.612845 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.612983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:19:15.616611 19363 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.616920 19363 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebff7a1acdd6441e964a1e36003c0730, State: Initialized, Role: FOLLOWER
I20260924 20:19:15.617223 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.606107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:40749 (local address 127.0.0.1:41912)
0924 20:19:15.607401 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.607420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.607538 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.607841 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.607845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.608049 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.608206 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.608219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.609238 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.609244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.610196 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.610208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.610811 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.610817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.610935 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.611433 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.611451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.612855 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.615387 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.615394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.615398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.615679 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.616036 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.616040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.616042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.616170 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.616673 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.616679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.616782 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.616903 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.617052 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179}
I20260924 20:19:15.617341 19363 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.617571 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.606224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.29.83.189:40749)
0924 20:19:15.606397 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.606405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.607520 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 20:19:15.607633 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.607638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.607695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.607796 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.608377 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.609098 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.610314 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.610657 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.610786 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.613012 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.613018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.613024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.613051 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.614924 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.615793 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.615797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.615799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.615904 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.616286 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.616290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.616293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.616561 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.616661 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.617107 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.617232 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:19:15.618083 19446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.618242 19446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.618667 19438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:15.619344 19363 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:19:15.628049 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.615339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.29.83.189:40749)
0924 20:19:15.615657 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.615667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.615681 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:19:15.615720 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.615724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.615778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.615857 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.616576 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.617397 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.618796 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.619138 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.619964 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.623144 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.623148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.623151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.623178 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.625618 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.626198 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.626201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.626203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.626301 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.626608 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.626611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.626613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.626857 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.627624 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.627699 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.627854 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
W20260924 20:19:15.628321 19457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:15.628492 19457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:15.624446 19295 master_runner.cc:186] Error getting master registration for 127.29.83.190:40101: OK, instance_id { permanent_uuid: "79b3262041454f66b871128a8064f340" instance_seqno: 1790281155092095 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:19:15.628844 19295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:19:15.630815 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.619282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36626)
0924 20:19:15.619487 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.619501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.619614 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.619858 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.619861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.620345 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.620439 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.620446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.621473 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.621480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.623614 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.623629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.624356 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.624360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.624436 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.625042 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.625058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.626808 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.629023 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.629030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.629034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.629283 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.629652 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.629655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.629658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.629749 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.630294 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.630297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.630400 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.630501 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.630639 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 20:19:15.630959 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.619313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.29.83.188:44581)
0924 20:19:15.619430 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.619434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.619698 (+ 264us) server_negotiation.cc:407] Connection header received
0924 20:19:15.619721 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.619723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.619753 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.619846 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.620603 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.621358 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.623763 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.624128 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.624237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.626970 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.626974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.626976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.627002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.628918 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.629412 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.629416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.629418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.629551 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.629879 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.629885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.629887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.630186 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.630286 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.630596 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.630749 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:19:15.632223 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:15.634971 19438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: Bootstrap starting.
I20260924 20:19:15.633855 19262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79b3262041454f66b871128a8064f340"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52308
I20260924 20:19:15.638145 19438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:15.639128 19438 log.cc:824] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:15.641357 19438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: No bootstrap required, opened a new log
I20260924 20:19:15.644761 19438 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.644987 19438 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb683d26da34e5bb3de7a2c7bc273ca, State: Initialized, Role: FOLLOWER
I20260924 20:19:15.645449 19438 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.646031 19460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.646178 19460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.646926 19438 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:19:15.654004 19471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:15.654191 19471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:15.651836 19370 master_runner.cc:186] Error getting master registration for 127.29.83.190:40101: OK, instance_id { permanent_uuid: "79b3262041454f66b871128a8064f340" instance_seqno: 1790281155092095 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:19:15.655597 19370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:19:15.735074 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } attempt: 4
Sep 24 20:19:15 dist-test-slave-52fr krb5kdc[19207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281155, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:19:15.744872 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.735873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.29.83.188:44581)
0924 20:19:15.736030 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.736036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.736099 (+ 63us) server_negotiation.cc:407] Connection header received
0924 20:19:15.736247 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.736250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.736313 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.736393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.736883 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.737521 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.738644 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.738878 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.738994 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.741429 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.741433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.741437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.741468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.743029 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.743542 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.743545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.743547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.743630 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.743974 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.743978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.743982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.744210 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.744318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.744595 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.744710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 20:19:15.744872 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.735794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:44581 (local address 127.0.0.1:36636)
0924 20:19:15.736028 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:15.736045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:15.736156 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:15.736409 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:15.736412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:15.736639 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:15.736755 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.736762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.737639 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.737643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.738526 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:15.738542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.739002 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.739005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:15.739094 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.739665 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:15.739680 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:15.741283 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:15.743163 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.743168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.743172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.743455 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.743762 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:15.743765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:15.743767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:15.743867 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:15.744318 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:15.744322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:15.744424 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 20:19:15.744531 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:19:15.744695 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 20:19:15.749415 19288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: Bootstrap starting.
I20260924 20:19:15.752681 19288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:15.753750 19288 log.cc:824] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:15.756291 19288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: No bootstrap required, opened a new log
I20260924 20:19:15.759644 19288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.759931 19288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b3262041454f66b871128a8064f340, State: Initialized, Role: FOLLOWER
I20260924 20:19:15.760295 19288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.760915 19474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.761195 19474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.761875 19288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:15.761909 19262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:19:15.762009 19262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:15.763800 19262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.765923 19337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79b3262041454f66b871128a8064f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ebff7a1acdd6441e964a1e36003c0730"
I20260924 20:19:15.766222 19337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:15.768178 19337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b3262041454f66b871128a8064f340 in term 1.
W20260924 20:19:15.768558 19485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:15.768642 19485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:15.768698 19237 leader_election.cc:302] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b3262041454f66b871128a8064f340, ebff7a1acdd6441e964a1e36003c0730; no voters:
I20260924 20:19:15.768985 19474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:15.769253 19474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [term 1 LEADER]: Becoming Leader. State: Replica: 79b3262041454f66b871128a8064f340, State: Running, Role: LEADER
I20260924 20:19:15.769676 19474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } }
I20260924 20:19:15.771200 19412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79b3262041454f66b871128a8064f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca"
I20260924 20:19:15.771485 19412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:15.771732 19474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b3262041454f66b871128a8064f340. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.771868 19474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:15.772775 19412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b3262041454f66b871128a8064f340 in term 1.
I20260924 20:19:15.773492 19262 leader_election.cc:288] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [CANDIDATE]: Term 1 election: Requested vote from peers ebff7a1acdd6441e964a1e36003c0730 (127.29.83.189:40749), bcb683d26da34e5bb3de7a2c7bc273ca (127.29.83.188:44581)
I20260924 20:19:15.779021 19220 master_runner.cc:263] No leader master found from master 79b3262041454f66b871128a8064f340
I20260924 20:19:15.779129 19220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:19:15.789788 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.776938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.29.83.190:40101)
0924 20:19:15.777055 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.777059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.779513 (+ 2454us) server_negotiation.cc:407] Connection header received
0924 20:19:15.779710 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.779713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.779763 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.779829 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.780934 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.781735 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.783021 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.783400 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.783485 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.785571 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.785576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.785579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.785611 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.787608 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.788198 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.788203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.788205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.788310 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.788642 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.788648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.788653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.788899 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.789028 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.789416 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.789567 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:19:15.794056 19489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:15.795234 19489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:15.802012 19412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [term 1 FOLLOWER]: Refusing update from remote peer 79b3262041454f66b871128a8064f340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:15.802012 19337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [term 1 FOLLOWER]: Refusing update from remote peer 79b3262041454f66b871128a8064f340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:15.803176 19474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:15.803768 19474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:15.808812 19446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b3262041454f66b871128a8064f340. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.808972 19446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.810577 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.776355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.29.83.188:44581)
0924 20:19:15.776497 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.776502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.786096 (+ 9594us) server_negotiation.cc:407] Connection header received
0924 20:19:15.786865 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.786870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.786947 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.787018 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.792032 (+ 5014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.792568 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.793401 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.793786 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.793892 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.806324 (+ 12432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.806328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.806331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.806375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.808218 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.808821 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.808827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.808832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.808928 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.809382 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.809388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.809393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.809678 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.809855 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.810277 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.810411 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:19:15.814709 19460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b3262041454f66b871128a8064f340. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.814841 19460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: This master's current role is: FOLLOWER
W20260924 20:19:15.817670 19316 tablet.cc:2369] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:19:15.819154 19391 tablet.cc:2369] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:19:15.819677 19498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.819844 19498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:15.819972 19497 mvcc.cc:205] Tried to move back new op lower bound from 7332991614156939264 to 7332991614036533248. Current Snapshot: MvccSnapshot[applied={T|T < 7332991614156939264}]
I20260924 20:19:15.820457 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:15.778053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.29.83.189:40749)
0924 20:19:15.778161 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.778166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.786496 (+ 8330us) server_negotiation.cc:407] Connection header received
0924 20:19:15.787200 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.787203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.787264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.787336 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.790419 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.791090 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.800324 (+ 9234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:15.800674 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:15.800775 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:15.809143 (+ 8368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:15.809146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:15.809149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:15.809180 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:15.816188 (+ 7008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.816752 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.816756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.816758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.816850 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:15.817127 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:15.817130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:15.817132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:15.817389 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:15.817539 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:15.820104 (+ 2565us) server_negotiation.cc:299] Negotiation successful
0924 20:19:15.820267 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 20:19:15.822613 19446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.822742 19446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.823868 19474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.824016 19474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:15.825361 19460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79b3262041454f66b871128a8064f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b3262041454f66b871128a8064f340" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 40101 } } peers { permanent_uuid: "ebff7a1acdd6441e964a1e36003c0730" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 40749 } } peers { permanent_uuid: "bcb683d26da34e5bb3de7a2c7bc273ca" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 44581 } } }
I20260924 20:19:15.825498 19460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:15.828601 19499 mvcc.cc:205] Tried to move back new op lower bound from 7332991614156939264 to 7332991614036533248. Current Snapshot: MvccSnapshot[applied={T|T < 7332991614156939264}]
I20260924 20:19:15.830034 19501 mvcc.cc:205] Tried to move back new op lower bound from 7332991614156939264 to 7332991614036533248. Current Snapshot: MvccSnapshot[applied={T|T < 7332991614156939264}]
I20260924 20:19:15.831370 19489 catalog_manager.cc:1347] Generated new cluster ID: 61734ce03ada486595b1ff053cbc0b7d
I20260924 20:19:15.831445 19489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:15.856247 19489 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:15.857460 19489 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:15.875592 19489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b3262041454f66b871128a8064f340: Generated new TSK 0
I20260924 20:19:15.876739 19489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:19:15.902391 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.900696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.29.83.189:40749)
0924 20:19:15.900839 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.900845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.901156 (+ 311us) server_negotiation.cc:407] Connection header received
0924 20:19:15.901372 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.901375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.901432 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.901487 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.902250 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41938: BlockingRecv error: recv got EOF from 127.0.0.1:41938 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:19:15.902436 19290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.900951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.29.83.190:40101)
0924 20:19:15.901085 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.901093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.901193 (+ 100us) server_negotiation.cc:407] Connection header received
0924 20:19:15.901424 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.901428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.901486 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.901541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.902273 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52328: BlockingRecv error: recv got EOF from 127.0.0.1:52328 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 20:19:15.903249 19442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.901539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.29.83.188:44581)
0924 20:19:15.901670 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.901676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.901834 (+ 158us) server_negotiation.cc:407] Connection header received
0924 20:19:15.902026 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.902030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.902082 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.902143 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.903116 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36646: BlockingRecv error: recv got EOF from 127.0.0.1:36646 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:19:15.906612 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.905441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.29.83.189:40749)
0924 20:19:15.905575 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.905579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.905711 (+ 132us) server_negotiation.cc:407] Connection header received
0924 20:19:15.905883 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.905886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.905955 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.906026 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.906493 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41944: BlockingRecv error: recv got EOF from 127.0.0.1:41944 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 20:19:15.910980 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.910067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.29.83.189:40749)
0924 20:19:15.910222 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.910226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.910237 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:19:15.910282 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.910284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.910331 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.910425 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.910845 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41956: BlockingRecv error: recv got EOF from 127.0.0.1:41956 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 20:19:15.916126 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.915015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.29.83.189:40749)
0924 20:19:15.915146 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.915150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.915299 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:19:15.915457 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.915460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.915518 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.915608 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.916020 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41970: BlockingRecv error: recv got EOF from 127.0.0.1:41970 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 20:19:15.923390 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.922276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.29.83.189:40749)
0924 20:19:15.922519 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.922523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.922545 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:19:15.922704 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.922707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.922764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.922857 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.923265 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260924 20:19:15.934787 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.933518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.29.83.189:40749)
0924 20:19:15.933670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.933675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.933852 (+ 177us) server_negotiation.cc:407] Connection header received
0924 20:19:15.934019 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.934023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.934086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.934193 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.934646 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41988: BlockingRecv error: recv got EOF from 127.0.0.1:41988 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 20:19:15.954196 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.952859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.29.83.189:40749)
0924 20:19:15.953011 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.953016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.953139 (+ 123us) server_negotiation.cc:407] Connection header received
0924 20:19:15.953249 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.953252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.953332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.953466 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.954058 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41990: BlockingRecv error: recv got EOF from 127.0.0.1:41990 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 20:19:15.989439 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:15.988346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.29.83.189:40749)
0924 20:19:15.988500 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:15.988503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:15.988603 (+ 100us) server_negotiation.cc:407] Connection header received
0924 20:19:15.988737 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:15.988739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:15.988787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:15.988890 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:15.989341 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41996: BlockingRecv error: recv got EOF from 127.0.0.1:41996 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 20:19:16.056977 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:16.055843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.29.83.189:40749)
0924 20:19:16.055999 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.056003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.056179 (+ 176us) server_negotiation.cc:407] Connection header received
0924 20:19:16.056242 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.056244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.056298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.056389 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:16.056873 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42000: BlockingRecv error: recv got EOF from 127.0.0.1:42000 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 20:19:16.188796 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:16.187468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.29.83.189:40749)
0924 20:19:16.187646 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.187652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.187868 (+ 216us) server_negotiation.cc:407] Connection header received
0924 20:19:16.187967 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.187970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.188020 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.188106 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:16.188641 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42014: BlockingRecv error: recv got EOF from 127.0.0.1:42014 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 20:19:16.448418 19445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:16.447115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.29.83.189:40749)
0924 20:19:16.447259 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.447263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.447430 (+ 167us) server_negotiation.cc:407] Connection header received
0924 20:19:16.447494 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.447497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.447549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.447640 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:16.448245 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42030: BlockingRecv error: recv got EOF from 127.0.0.1:42030 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:19:16.630685 19457 catalog_manager.cc:1259] Loaded cluster ID: 61734ce03ada486595b1ff053cbc0b7d
I20260924 20:19:16.630790 19457 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: loading cluster ID for follower catalog manager: success
I20260924 20:19:16.633816 19457 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: acquiring CA information for follower catalog manager: success
I20260924 20:19:16.635792 19457 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebff7a1acdd6441e964a1e36003c0730: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:19:16.656157 19471 catalog_manager.cc:1259] Loaded cluster ID: 61734ce03ada486595b1ff053cbc0b7d
I20260924 20:19:16.656248 19471 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: loading cluster ID for follower catalog manager: success
I20260924 20:19:16.658841 19471 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: acquiring CA information for follower catalog manager: success
I20260924 20:19:16.660689 19471 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcb683d26da34e5bb3de7a2c7bc273ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:19:16.967677 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:16.962895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.29.83.189:40749)
0924 20:19:16.963172 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.963179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.963289 (+ 110us) server_negotiation.cc:407] Connection header received
0924 20:19:16.963630 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.963634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.963770 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.963890 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:16.964259 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.964826 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.965826 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.966087 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.966193 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:16.966575 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:16.967045 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:16.967161 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:16.967388 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 20:19:16.967470 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260924 20:19:16.975435 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:16.971541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.29.83.188:44581)
0924 20:19:16.971823 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.971829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.971846 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:19:16.971942 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.971944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.972064 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.972192 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:16.972439 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.972964 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.973917 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.974147 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.974260 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:16.974578 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:16.975026 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:16.975169 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:16.975205 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 20:19:16.975266 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":66,"threads_started":1}
I20260924 20:19:16.983153 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:16.978872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.29.83.189:40749)
0924 20:19:16.979013 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.979018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.979280 (+ 262us) server_negotiation.cc:407] Connection header received
0924 20:19:16.979508 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.979512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.979589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.979685 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:16.979971 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.980743 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.981807 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.982037 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.982153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:16.982577 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:16.982655 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:16.982722 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:16.982868 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:19:16.982929 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:19:16.983644 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:16.979367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.29.83.188:44581)
0924 20:19:16.979492 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.979501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.979957 (+ 456us) server_negotiation.cc:407] Connection header received
0924 20:19:16.980129 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.980134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.980191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.980277 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:16.980516 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.981182 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.982474 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.982747 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.982858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:16.983139 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:16.983214 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:16.983294 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:16.983459 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 20:19:16.983516 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:19:16.984714 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:16.978969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.29.83.190:40101)
0924 20:19:16.979240 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:16.979245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:16.979386 (+ 141us) server_negotiation.cc:407] Connection header received
0924 20:19:16.979641 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:16.979647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:16.979832 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:16.979970 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:16.980374 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.981095 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.982933 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:16.983393 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:16.983491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:16.983939 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:16.984322 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:16.984414 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:16.984547 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 20:19:16.984604 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260924 20:19:16.988490 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19220
I20260924 20:19:16.995250 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19295
I20260924 20:19:17.001461 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19370
2026-09-24T20:19:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3287 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:17 dist-test-slave-52fr krb5kdc[19587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:17 dist-test-slave-52fr krb5kdc[19587](info): set up 2 sockets
Sep 24 20:19:17 dist-test-slave-52fr krb5kdc[19587](info): commencing operation
krb5kdc: starting...
W20260924 20:19:18.054190 30030 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/test-user.keytab.
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.83.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
Entry for principal kudu/127.29.83.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab.
I20260924 20:19:18.186269 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45249
--webserver_interface=127.29.83.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:18.305709 19600 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:18.306051 19600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:18.306082 19600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:18.306125 19600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:18.310242 19600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:18.310315 19600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:18.310335 19600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:18.310352 19600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:18.310367 19600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:18.315744 19600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.190.keytab
--principal=kudu/127.29.83.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.190:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.83.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:18.316800 19600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:18.317857 19600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:18.326766 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:18.326761 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:18.326918 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:18.327091 19600 server_base.cc:1034] running on GCE node
I20260924 20:19:18.327931 19600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:18.329255 19600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:18.330508 19600 hybrid_clock.cc:648] HybridClock initialized: now 1790281158330476 us; error 48 us; skew 500 ppm
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:18.333567 19600 init.cc:377] Logged in from keytab as kudu/127.29.83.190@KRBTEST.COM (short username kudu)
I20260924 20:19:18.334650 19600 webserver.cc:460] Webserver started at http://127.29.83.190:43747/ using document root <none> and password file <none>
I20260924 20:19:18.335243 19600 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:18.335336 19600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:18.335567 19600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:18.337841 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data/instance:
uuid: "494b5c6f24164828a59befbff37d6989"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "0d247425f8d73cdbecd4cab5167bcd87"
server_key_iv: "bd90068edadad025d1e9543eb8912794"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.338467 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance:
uuid: "494b5c6f24164828a59befbff37d6989"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "0d247425f8d73cdbecd4cab5167bcd87"
server_key_iv: "bd90068edadad025d1e9543eb8912794"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.342481 19600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:19:18.345191 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:18.346733 19600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:18.346899 19600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal
uuid: "494b5c6f24164828a59befbff37d6989"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "0d247425f8d73cdbecd4cab5167bcd87"
server_key_iv: "bd90068edadad025d1e9543eb8912794"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.347026 19600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:18.376978 19600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:18.380450 19600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:18.380658 19600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:18.388381 19600 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.190:45249
I20260924 20:19:18.388670 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.190:45249 every 8 connection(s)
I20260924 20:19:18.389457 19600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:18.392421 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:18.393409 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19600
I20260924 20:19:18.393641 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-0/wal/instance
I20260924 20:19:18.393957 30030 external_mini_cluster.cc:1428] Setting key 270e5e0fd2fd16f1c6fee09f3c51e7ad
I20260924 20:19:18.396457 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.83.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.189@KRBTEST.COM" created.
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:18.412444 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.398927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45249 (local address 127.0.0.1:49954)
0924 20:19:18.399512 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.399547 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.400376 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.401042 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.401047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.401490 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.401821 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.401841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.403002 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.403005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.403739 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.403747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.404392 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.404396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.404505 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.405109 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.405126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.406891 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.409207 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.409215 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.409227 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.409467 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.409824 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.409829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.409831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.409950 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.410585 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.410591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.410738 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.410913 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.411359 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":106,"thread_start_us":139,"threads_started":1}
I20260924 20:19:18.412444 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.398926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.29.83.190:45249)
0924 20:19:18.399372 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.399392 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.399662 (+ 270us) server_negotiation.cc:407] Connection header received
0924 20:19:18.400676 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.400681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.400863 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.401047 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.401966 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.402888 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.403854 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.404238 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.404407 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.407177 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.407183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.407196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.407219 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.409067 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.409591 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.409597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.409598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.409711 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.410045 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.410048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.410050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.410463 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.410607 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.410974 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.411306 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20260924 20:19:18.415795 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.416604 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.416314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55322)
0924 20:19:18.416449 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.416527 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 20:19:18.416848 19616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
I20260924 20:19:18.417868 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.417592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55334)
0924 20:19:18.417702 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.417792 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:19:18.418040 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:41567: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
Entry for principal kudu/127.29.83.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab.
I20260924 20:19:18.421335 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:41567
--webserver_interface=127.29.83.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:19:18.467327 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } attempt: 1
I20260924 20:19:18.468339 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.467994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55350)
0924 20:19:18.468163 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.468233 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 20:19:18.469077 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.468857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55362)
0924 20:19:18.468967 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.469019 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:19:18.469209 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:41567: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
I20260924 20:19:18.537397 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } attempt: 2
I20260924 20:19:18.538344 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.538021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55364)
0924 20:19:18.538198 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.538265 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:19:18.539068 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.538855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55370)
0924 20:19:18.538960 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.539010 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:19:18.539211 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.189:41567: Network error: Client connection negotiation failed: client connection to 127.29.83.189:41567: connect: Connection refused (error 111)
W20260924 20:19:18.540462 19675 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:18.540776 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:18.540827 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:18.540917 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:18.545225 19675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:18.545287 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:18.545308 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:18.545326 19675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:18.545379 19675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:18.551107 19675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.189.keytab
--principal=kudu/127.29.83.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.189:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.83.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:18.552327 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:18.553396 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:18.562233 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:18.562469 19675 server_base.cc:1034] running on GCE node
W20260924 20:19:18.562229 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:18.562305 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:18.563194 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:18.564370 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:18.565546 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790281158565512 us; error 46 us; skew 500 ppm
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:18.568540 19675 init.cc:377] Logged in from keytab as kudu/127.29.83.189@KRBTEST.COM (short username kudu)
I20260924 20:19:18.569473 19675 webserver.cc:460] Webserver started at http://127.29.83.189:41643/ using document root <none> and password file <none>
I20260924 20:19:18.570086 19675 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:18.570150 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:18.570328 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:18.572463 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data/instance:
uuid: "ffccf56d7e0246d3baf3fdbc11138b32"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "40e4ab7600a7cf723e5d0f5ad9123a88"
server_key_iv: "9270219748723d86ed85f1d893b7d884"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.572990 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal/instance:
uuid: "ffccf56d7e0246d3baf3fdbc11138b32"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "40e4ab7600a7cf723e5d0f5ad9123a88"
server_key_iv: "9270219748723d86ed85f1d893b7d884"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.576993 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:19:18.579650 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:18.581002 19675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:19:18.581151 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal
uuid: "ffccf56d7e0246d3baf3fdbc11138b32"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "40e4ab7600a7cf723e5d0f5ad9123a88"
server_key_iv: "9270219748723d86ed85f1d893b7d884"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.581287 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:18.602475 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:18.606302 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:18.606524 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:18.614459 19675 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.189:41567
I20260924 20:19:18.614790 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.189:41567 every 8 connection(s)
I20260924 20:19:18.615510 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/data/info.pb
I20260924 20:19:18.618368 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19675
I20260924 20:19:18.618495 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-1/wal/instance
I20260924 20:19:18.618631 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:18.618840 30030 external_mini_cluster.cc:1428] Setting key 6ace815c2a8de55814772570f33810a2
I20260924 20:19:18.623917 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.83.188@KRBTEST.COM; defaulting to no policy
I20260924 20:19:18.630405 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.83.188@KRBTEST.COM" created.
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:18.648717 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.631004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55380)
0924 20:19:18.631243 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.631261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.631354 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.632302 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.632308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.632534 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.632692 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.632703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.633992 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.633996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.634797 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.634808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.635473 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.635477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.635598 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.641712 (+ 6114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.641728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.643373 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.646470 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.646473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.646474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.646681 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.647136 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.647139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.647141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.647242 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.648064 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.648067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.648182 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.648369 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.648530 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 20:19:18.649086 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.626806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.29.83.190:45249)
0924 20:19:18.626992 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.626999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.627891 (+ 892us) server_negotiation.cc:407] Connection header received
0924 20:19:18.629128 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.629131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.629189 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.629313 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.631115 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.631986 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.633603 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.634048 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.634176 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.644233 (+ 10057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.644240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.644246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.644292 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.646153 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.646858 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.646862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.646864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.646952 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.647298 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.647301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.647303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.647509 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.647619 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.648803 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.648936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:19:18.649735 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.627186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45249 (local address 127.0.0.1:49962)
0924 20:19:18.627708 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.627774 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.628880 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.629594 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.629602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.630267 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.630895 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.630911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.632160 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.632166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.633468 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.633479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.634235 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.634239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.634458 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.641929 (+ 7471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.641974 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.644017 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.646332 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.646348 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.646365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.646720 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.647056 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.647059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.647061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.647192 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.647619 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.647628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.647912 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.648101 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.648543 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260924 20:19:18.649740 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.631105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.29.83.189:41567)
0924 20:19:18.631621 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.631632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.631658 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:19:18.631701 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.631706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.631913 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.632269 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.632845 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.633887 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.634915 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.635351 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.635467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.643661 (+ 8194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.643670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.643687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.643721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.646276 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.646885 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.646892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.646894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.647023 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.647338 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.647343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.647344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.647952 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.648104 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.648489 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.648644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":418,"spinlock_wait_cycles":2304,"thread_start_us":185,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
Entry for principal kudu/127.29.83.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab.
I20260924 20:19:18.650924 30030 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
/tmp/dist-test-taskrGCYXn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:46363
--webserver_interface=127.29.83.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.83.148:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:19:18.651368 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.652029 19668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.652694 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.652463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51714)
0924 20:19:18.652577 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.652635 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 20:19:18.653448 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.653128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51718)
0924 20:19:18.653311 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.653373 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 20:19:18.653646 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:19:18.661151 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.652113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.29.83.189:41567)
0924 20:19:18.652248 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.652256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.652271 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:19:18.652314 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.652319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.652373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.652463 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.653068 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.653728 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.654668 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.655130 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.655254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.657600 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.657603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.657605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.657626 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.659217 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.659743 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.659745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.659746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.659820 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.660118 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.660120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.660121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.660306 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.660390 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.660733 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.660977 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":43}
I20260924 20:19:18.661155 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.651887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55388)
0924 20:19:18.652072 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.652087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.652221 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.652495 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.652499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.652710 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.652956 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.652963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.653892 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.653897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.654547 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.654556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.655235 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.655237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.655349 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.655857 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.655869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.657443 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.659329 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.659335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.659340 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.659625 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.659909 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.659914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.659918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.660026 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.660404 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.660409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.660532 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.660664 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.660864 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 20:19:18.662190 19743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.663007 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.662725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51728)
0924 20:19:18.662858 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.662944 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 20:19:18.663712 19693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
I20260924 20:19:18.664609 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.664404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51740)
0924 20:19:18.664506 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.664554 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 20:19:18.664845 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
I20260924 20:19:18.684834 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 1
I20260924 20:19:18.685883 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.685479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51754)
0924 20:19:18.685661 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.685749 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 20:19:18.686787 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.686550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51768)
0924 20:19:18.686669 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.686722 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 20:19:18.687004 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
I20260924 20:19:18.714131 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 1
I20260924 20:19:18.715111 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.714781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51780)
0924 20:19:18.714961 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.715031 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 20:19:18.715870 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.715645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51782)
0924 20:19:18.715762 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.715814 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:19:18.715999 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
I20260924 20:19:18.762162 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 2
I20260924 20:19:18.763023 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.762707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51784)
0924 20:19:18.762879 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.762942 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:19:18.763818 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.763581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51798)
0924 20:19:18.763703 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.763756 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:19:18.763948 19668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
W20260924 20:19:18.770782 19749 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:19:18.771149 19749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:18.771207 19749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:18.771309 19749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:18.775645 19749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:18.775733 19749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:18.775770 19749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:18.775820 19749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:18.775857 19749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:18.781420 19749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.83.148:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.83.190:45249,127.29.83.189:41567,127.29.83.188:46363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/krb5kdc/kudu_127.29.83.188.keytab
--principal=kudu/127.29.83.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.83.188:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.83.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:18.782619 19749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:18.783710 19749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:19:18.784150 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 2
I20260924 20:19:18.784981 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.784696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51810)
0924 20:19:18.784840 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.784905 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 20:19:18.785794 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.785523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51822)
0924 20:19:18.785624 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.785672 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 20:19:18.785926 19743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.83.188:46363: Network error: Client connection negotiation failed: client connection to 127.29.83.188:46363: connect: Connection refused (error 111)
W20260924 20:19:18.792887 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:18.792892 19749 server_base.cc:1034] running on GCE node
W20260924 20:19:18.792853 19755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:18.792912 19756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:18.793668 19749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:18.794837 19749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:18.795996 19749 hybrid_clock.cc:648] HybridClock initialized: now 1790281158795966 us; error 44 us; skew 500 ppm
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:18.798915 19749 init.cc:377] Logged in from keytab as kudu/127.29.83.188@KRBTEST.COM (short username kudu)
I20260924 20:19:18.799883 19749 webserver.cc:460] Webserver started at http://127.29.83.188:33501/ using document root <none> and password file <none>
I20260924 20:19:18.800483 19749 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:18.800549 19749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:18.800772 19749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:18.803071 19749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data/instance:
uuid: "2927826fb15b43de8648461cd321add9"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "da3187672c6ee964dd6608d426bfddfc"
server_key_iv: "cd652707aebd513750690de987c9e59e"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.803675 19749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal/instance:
uuid: "2927826fb15b43de8648461cd321add9"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "da3187672c6ee964dd6608d426bfddfc"
server_key_iv: "cd652707aebd513750690de987c9e59e"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.807585 19749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:19:18.810177 19765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:18.811510 19749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:19:18.811645 19749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data,/tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal
uuid: "2927826fb15b43de8648461cd321add9"
format_stamp: "Formatted at 2026-09-24 20:19:18 on dist-test-slave-52fr"
server_key: "da3187672c6ee964dd6608d426bfddfc"
server_key_iv: "cd652707aebd513750690de987c9e59e"
server_key_version: "encryptionkey@0"
I20260924 20:19:18.811756 19749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:18.836925 19749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:18.840267 19749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:18.840471 19749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:18.847899 19749 rpc_server.cc:307] RPC server started. Bound to: 127.29.83.188:46363
I20260924 20:19:18.848198 19817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.83.188:46363 every 8 connection(s)
I20260924 20:19:18.849015 19749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/data/info.pb
I20260924 20:19:18.852267 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:18.856197 19818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.859081 30030 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu as pid 19749
I20260924 20:19:18.859211 30030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrGCYXn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280784619584-30030-0/minicluster-data/master-2/wal/instance
I20260924 20:19:18.859510 30030 external_mini_cluster.cc:1428] Setting key f01bad4d0644c34ef74c22fe0c95f7d6
I20260924 20:19:18.863106 19668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 3
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.190@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:19:18.872792 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.858216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.29.83.190:45249)
0924 20:19:18.858350 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.858354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.859344 (+ 990us) server_negotiation.cc:407] Connection header received
0924 20:19:18.860727 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.860733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.860800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.860906 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.862376 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.862887 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.864251 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.864563 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.864672 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.867931 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.867938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.867942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.867977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.870160 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.870807 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.870811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.870813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.870905 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871355 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.871362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.871366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.871713 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.871813 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.872486 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.872614 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:19:18.873170 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.863537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51848)
0924 20:19:18.863705 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.863718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.863793 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.864146 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.864152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.864367 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.864468 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.864475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.865368 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.865380 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.866176 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.866188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.866895 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.866899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.866995 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.867518 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.867530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.868833 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.871443 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.871449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.871798 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.872205 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.872208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.872210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.872303 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.872715 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.872719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.872808 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.872889 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.873010 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 20:19:18.873777 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.860279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.29.83.188:46363)
0924 20:19:18.860685 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.860696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.860735 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:19:18.860942 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.860947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.861182 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.861480 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.862370 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.863254 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.864385 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.864937 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.865041 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.868472 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.868479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.868494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.868523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.870873 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871307 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.871313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.871315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.871410 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871745 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.871751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.871756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.872198 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.872333 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.872774 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.872930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":12288,"thread_start_us":193,"threads_started":1}
I20260924 20:19:18.873792 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.858580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.190:45249 (local address 127.0.0.1:49972)
0924 20:19:18.859160 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.859229 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.860394 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.860982 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.860992 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.861582 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.862212 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.862232 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.863001 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.863007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.864094 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.864106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.864713 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.864717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.864891 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.865917 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.865937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.867699 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.870282 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.870290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.870302 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.870686 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.871025 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.871032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.871224 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.871824 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.871835 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.872187 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.872438 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.872780 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":27,"thread_start_us":170,"threads_started":1}
I20260924 20:19:18.874658 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.863612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.29.83.188:46363)
0924 20:19:18.863922 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.863930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.863942 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:19:18.863975 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.863978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.864017 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.864103 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.864588 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.865243 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.866344 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.866771 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.866856 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.868979 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.868982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.868984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.869005 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.871309 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.871976 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.871979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.871981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.872102 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.872405 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.872409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.872412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.872597 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.873773 (+ 1176us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.874395 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.874513 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260924 20:19:18.876068 19818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:19:18.877353 19743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } attempt: 3
I20260924 20:19:18.879359 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: Bootstrap starting.
I20260924 20:19:18.882323 19668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: Neither blocks nor log segments found. Creating new log.
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.190@KRBTEST.COM
I20260924 20:19:18.883359 19668 log.cc:824] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:18.886545 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: No bootstrap required, opened a new log
I20260924 20:19:18.887917 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.876645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.189:41567 (local address 127.0.0.1:55390)
0924 20:19:18.876754 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.876770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.876856 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.877248 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.877252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.877524 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.878199 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.878205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.879291 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.879295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.880060 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.880071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.880753 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.880757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.880858 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.881353 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.881367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.883592 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.886297 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.886301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.886303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.886563 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.886865 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.886868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.886870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.886954 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.887392 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.887395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.887482 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.887577 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.887727 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 20:19:18.888666 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.877035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.29.83.190:45249)
0924 20:19:18.877149 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.877154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.877169 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:19:18.877293 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.877296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.877338 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.877526 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.878481 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.879212 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.880100 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.880531 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.880646 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.883742 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.883748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.883751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.883779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.886228 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.886857 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.886862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.886866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.886964 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.887282 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.887286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.887288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.887498 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.887653 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.888368 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.888501 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:19:18.888798 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.876839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.29.83.189:41567)
0924 20:19:18.876987 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.876992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.877006 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:19:18.877082 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.877087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.877139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.877240 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.878320 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.879130 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.880184 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.880631 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.880727 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.883798 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.883802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.883805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.883836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.886173 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.886677 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.886680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.886682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.886769 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.887042 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.887045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.887047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.887278 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.887357 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.888527 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.888653 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:19:18.889510 19818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.189@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
I20260924 20:19:18.892671 19668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.892961 19668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494b5c6f24164828a59befbff37d6989, State: Initialized, Role: FOLLOWER
I20260924 20:19:18.893358 19668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.893885 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.877754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51862)
0924 20:19:18.877944 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.877956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.878022 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.884375 (+ 6353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.884380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.884820 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.884975 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.884987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.886294 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.886297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.887126 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.887134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.887746 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.887750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.887851 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.888345 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.888361 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.889998 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.892106 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.892109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.892111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.892357 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.892668 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.892671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.892673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.892759 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.893353 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.893356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.893456 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.893540 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.893686 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 20:19:18.895033 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.877844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.29.83.188:46363)
0924 20:19:18.884103 (+ 6259us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.884110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.884126 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:19:18.884165 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.884170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.884222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.885164 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.885276 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.886159 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.887254 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.887609 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.888069 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.890146 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.890151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.890153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.890177 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.891976 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.892480 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.892484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.892485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.892569 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.892887 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.892890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.892891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.893236 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.894666 (+ 1430us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.894749 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.894886 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6149}
I20260924 20:19:18.895045 19668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.83.188@KRBTEST.COM for kudu/127.29.83.188@KRBTEST.COM
Sep 24 20:19:18 dist-test-slave-52fr krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.83.189@KRBTEST.COM
I20260924 20:19:18.900030 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.890053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.83.188:46363 (local address 127.0.0.1:51872)
0924 20:19:18.890217 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:18.890231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:18.890315 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:18.890915 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:18.890917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:18.891098 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:18.891200 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.891211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.892557 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.892559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.893323 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:18.893333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.894019 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.894022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:18.894102 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.894596 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:18.894610 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:18.896231 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:18.898382 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.898387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.898390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.898624 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.898894 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:18.898898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:18.898902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:18.898982 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:18.899497 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:18.899501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:18.899607 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 20:19:18.899718 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:19:18.899869 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 20:19:18.900630 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.890096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.29.83.188:46363)
0924 20:19:18.890208 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.890226 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.890276 (+ 50us) server_negotiation.cc:407] Connection header received
0924 20:19:18.890680 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.890683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.890735 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.890812 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.891659 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.892449 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.893434 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.893825 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.893896 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.896390 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.896397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.896401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.896428 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.898261 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.898719 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.898722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.898726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.898795 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.899087 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.899090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.899093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.899348 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.899457 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.900338 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.900471 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:19:18.900930 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: Bootstrap starting.
W20260924 20:19:18.901121 19836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:18.901183 19836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:18.903723 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: Bootstrap starting.
I20260924 20:19:18.903986 19743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:18.900655 19600 master_runner.cc:186] Error getting master registration for 127.29.83.189:41567: OK, instance_id { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" instance_seqno: 1790281158612295 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:19:18.904752 19600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:19:18.904968 19743 log.cc:824] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:18.906659 19818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:18.907625 19818 log.cc:824] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:18.907753 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.890636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.29.83.189:41567)
0924 20:19:18.891375 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.891378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.891387 (+ 9us) server_negotiation.cc:407] Connection header received
0924 20:19:18.891414 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.891416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.891448 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.891538 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.892287 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.893027 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.894159 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.894827 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.894905 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.898625 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.898629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.898632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.898662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.905432 (+ 6770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.906134 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.906140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.906142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.906243 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.906591 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.906595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.906597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.906857 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.907055 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.907431 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.907570 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
I20260924 20:19:18.908274 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: No bootstrap required, opened a new log
I20260924 20:19:18.909842 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: No bootstrap required, opened a new log
I20260924 20:19:18.911640 30030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:19:18.912945 19642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "494b5c6f24164828a59befbff37d6989"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986
I20260924 20:19:18.913250 19642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:19:18.913398 19642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:18.913980 19818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.914186 19818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2927826fb15b43de8648461cd321add9, State: Initialized, Role: FOLLOWER
I20260924 20:19:18.914305 19743 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.914492 19818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.914646 19743 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccf56d7e0246d3baf3fdbc11138b32, State: Initialized, Role: FOLLOWER
I20260924 20:19:18.914973 19840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.915109 19840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.915046 19743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.915460 19642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.915773 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.915894 19818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:18.915906 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.916815 19743 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:18.916898 19642 leader_election.cc:288] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [CANDIDATE]: Term 1 election: Requested vote from peers ffccf56d7e0246d3baf3fdbc11138b32 (127.29.83.189:41567), 2927826fb15b43de8648461cd321add9 (127.29.83.188:46363)
I20260924 20:19:18.917022 19717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "494b5c6f24164828a59befbff37d6989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ffccf56d7e0246d3baf3fdbc11138b32"
I20260924 20:19:18.917258 19717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:18.917237 19825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.917361 19825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.918794 19717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494b5c6f24164828a59befbff37d6989 in term 1.
I20260924 20:19:18.919554 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "494b5c6f24164828a59befbff37d6989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2927826fb15b43de8648461cd321add9"
I20260924 20:19:18.919780 19792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:18.921123 19792 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494b5c6f24164828a59befbff37d6989 in term 1.
I20260924 20:19:18.921627 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2927826fb15b43de8648461cd321add9, 494b5c6f24164828a59befbff37d6989, ffccf56d7e0246d3baf3fdbc11138b32; no voters:
I20260924 20:19:18.922026 19825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:18.922251 19825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [term 1 LEADER]: Becoming Leader. State: Replica: 494b5c6f24164828a59befbff37d6989, State: Running, Role: LEADER
I20260924 20:19:18.922607 19825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } }
I20260924 20:19:18.924636 19825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494b5c6f24164828a59befbff37d6989. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.924793 19825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: This master's current role is: LEADER
W20260924 20:19:18.927330 19858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:18.927410 19858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 20:19:18.929364 19869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:18.929498 19869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:18.929633 19675 master_runner.cc:263] No leader master found from master ffccf56d7e0246d3baf3fdbc11138b32
I20260924 20:19:18.929715 19675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:19:18.935714 19873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:18.936774 19873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:18.950031 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.925120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.29.83.189:41567)
0924 20:19:18.925239 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.925244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.936305 (+ 11061us) server_negotiation.cc:407] Connection header received
0924 20:19:18.936543 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.936549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.936605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.936659 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.937838 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.938643 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.939869 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.940227 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.940340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.946134 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.946138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.946141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.946182 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.947981 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.948518 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.948524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.948528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.948621 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.948953 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.948957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.948959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.949220 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.949335 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.949689 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.949858 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 20:19:18.950600 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.924446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.29.83.188:46363)
0924 20:19:18.924544 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.924550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.940061 (+ 15511us) server_negotiation.cc:407] Connection header received
0924 20:19:18.940173 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.940176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.940236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.940321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.941355 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.942166 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.943290 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.943643 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.943738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.946859 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.946865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.946870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.946902 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.948591 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.949071 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.949075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.949077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.949156 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.949463 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.949467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.949469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.949700 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.949835 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.950257 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.950393 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 20:19:18.959667 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:18.924261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.29.83.190:45249)
0924 20:19:18.924380 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:18.924385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:18.950067 (+ 25682us) server_negotiation.cc:407] Connection header received
0924 20:19:18.950889 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:18.950892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:18.950949 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:18.951026 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:18.951868 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.952379 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.953483 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:18.953847 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:18.953949 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:18.955745 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:18.955749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:18.955752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:18.955781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:18.957496 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.958021 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.958025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.958027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.958121 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:18.958402 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:18.958406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:18.958407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:18.958614 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:18.958730 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:18.959371 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 20:19:18.959488 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:19:18.965261 19717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [term 1 FOLLOWER]: Refusing update from remote peer 494b5c6f24164828a59befbff37d6989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:18.965291 19792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [term 1 FOLLOWER]: Refusing update from remote peer 494b5c6f24164828a59befbff37d6989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:18.966001 19863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:18.966468 19825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:18.970876 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494b5c6f24164828a59befbff37d6989. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.971012 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.977730 19840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494b5c6f24164828a59befbff37d6989. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.977900 19840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.982151 19825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.982295 19825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:18.982535 19876 mvcc.cc:205] Tried to move back new op lower bound from 7332991627115016192 to 7332991626951163904. Current Snapshot: MvccSnapshot[applied={T|T < 7332991627115016192}]
I20260924 20:19:18.985087 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:18.985212 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:18.986621 19877 mvcc.cc:205] Tried to move back new op lower bound from 7332991627115016192 to 7332991626951163904. Current Snapshot: MvccSnapshot[applied={T|T < 7332991627115016192}]
I20260924 20:19:18.987700 19873 catalog_manager.cc:1347] Generated new cluster ID: 3c4a3e113884491cbe8dc08c99ad3fb6
I20260924 20:19:18.987767 19873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:19.001067 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:19.001235 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:19.002573 19840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494b5c6f24164828a59befbff37d6989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494b5c6f24164828a59befbff37d6989" member_type: VOTER last_known_addr { host: "127.29.83.190" port: 45249 } } peers { permanent_uuid: "ffccf56d7e0246d3baf3fdbc11138b32" member_type: VOTER last_known_addr { host: "127.29.83.189" port: 41567 } } peers { permanent_uuid: "2927826fb15b43de8648461cd321add9" member_type: VOTER last_known_addr { host: "127.29.83.188" port: 46363 } } }
I20260924 20:19:19.002709 19840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:19:19.002837 19879 mvcc.cc:205] Tried to move back new op lower bound from 7332991627115016192 to 7332991626951163904. Current Snapshot: MvccSnapshot[applied={T|T < 7332991627115016192}]
I20260924 20:19:19.027930 19873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:19.029263 19873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:19.066844 19873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494b5c6f24164828a59befbff37d6989: Generated new TSK 0
I20260924 20:19:19.067847 19873 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:19:19.106105 19823 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.102657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.29.83.188:46363)
0924 20:19:19.102782 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.102787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.104902 (+ 2115us) server_negotiation.cc:407] Connection header received
0924 20:19:19.105109 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.105113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.105163 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.105220 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.105956 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51892: BlockingRecv error: recv got EOF from 127.0.0.1:51892 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 20:19:19.107923 19671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.103457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.29.83.190:45249)
0924 20:19:19.103574 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.103578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.106682 (+ 3104us) server_negotiation.cc:407] Connection header received
0924 20:19:19.106901 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.106904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.106960 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.107011 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.107777 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50000: BlockingRecv error: recv got EOF from 127.0.0.1:50000 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:19:19.109616 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.104262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.29.83.189:41567)
0924 20:19:19.104371 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.104375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.108361 (+ 3986us) server_negotiation.cc:407] Connection header received
0924 20:19:19.108745 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.108749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.108801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.108856 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.109498 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55406: BlockingRecv error: recv got EOF from 127.0.0.1:55406 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 20:19:19.114454 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.112714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.29.83.189:41567)
0924 20:19:19.112824 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.112828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.113278 (+ 450us) server_negotiation.cc:407] Connection header received
0924 20:19:19.113531 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.113535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.113630 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.113695 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.114337 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55418: BlockingRecv error: recv got EOF from 127.0.0.1:55418 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 20:19:19.119699 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.118160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.29.83.189:41567)
0924 20:19:19.118269 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.118275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.118761 (+ 486us) server_negotiation.cc:407] Connection header received
0924 20:19:19.118965 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.118968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.119020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.119083 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.119595 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55428: BlockingRecv error: recv got EOF from 127.0.0.1:55428 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:19:19.128244 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.126240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.29.83.189:41567)
0924 20:19:19.126386 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.126392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.127131 (+ 739us) server_negotiation.cc:407] Connection header received
0924 20:19:19.127371 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.127375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.127435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.127490 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.128115 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55438: BlockingRecv error: recv got EOF from 127.0.0.1:55438 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 20:19:19.136935 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.135148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.29.83.189:41567)
0924 20:19:19.135316 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.135321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.135906 (+ 585us) server_negotiation.cc:407] Connection header received
0924 20:19:19.136141 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.136145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.136198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.136252 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.136821 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55444: BlockingRecv error: recv got EOF from 127.0.0.1:55444 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 20:19:19.148854 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.147659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.29.83.189:41567)
0924 20:19:19.147808 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.147813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.147987 (+ 174us) server_negotiation.cc:407] Connection header received
0924 20:19:19.148110 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.148113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.148170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.148268 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.148722 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55450: BlockingRecv error: recv got EOF from 127.0.0.1:55450 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 20:19:19.168035 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.166908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.29.83.189:41567)
0924 20:19:19.167194 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.167198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.167214 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:19:19.167309 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.167311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.167357 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.167440 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.167907 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55454: BlockingRecv error: recv got EOF from 127.0.0.1:55454 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
W20260924 20:19:19.203466 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.202169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.29.83.189:41567)
0924 20:19:19.202339 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.202344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.202489 (+ 145us) server_negotiation.cc:407] Connection header received
0924 20:19:19.202666 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.202669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.202721 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.202808 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.203316 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55468: BlockingRecv error: recv got EOF from 127.0.0.1:55468 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 20:19:19.270861 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.269629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.29.83.189:41567)
0924 20:19:19.269826 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.269831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.269981 (+ 150us) server_negotiation.cc:407] Connection header received
0924 20:19:19.270150 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.270153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.270213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.270300 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.270742 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55472: BlockingRecv error: recv got EOF from 127.0.0.1:55472 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 20:19:19.403380 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.400927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.29.83.189:41567)
0924 20:19:19.401092 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.401097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.402459 (+ 1362us) server_negotiation.cc:407] Connection header received
0924 20:19:19.402641 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.402644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.402695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.402789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.403262 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55474: BlockingRecv error: recv got EOF from 127.0.0.1:55474 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 20:19:19.663172 19747 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:19:19.661945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.29.83.189:41567)
0924 20:19:19.662103 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:19.662109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:19.662214 (+ 105us) server_negotiation.cc:407] Connection header received
0924 20:19:19.662423 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:19.662427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:19.662482 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:19.662570 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:19.663052 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55490: BlockingRecv error: recv got EOF from 127.0.0.1:55490 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 20:19:19.928730 19858 catalog_manager.cc:1259] Loaded cluster ID: 3c4a3e113884491cbe8dc08c99ad3fb6
I20260924 20:19:19.928813 19858 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: loading cluster ID for follower catalog manager: success
I20260924 20:19:19.930478 19858 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: acquiring CA information for follower catalog manager: success
I20260924 20:19:19.930721 19869 catalog_manager.cc:1259] Loaded cluster ID: 3c4a3e113884491cbe8dc08c99ad3fb6
I20260924 20:19:19.930796 19869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: loading cluster ID for follower catalog manager: success
I20260924 20:19:19.931708 19858 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ffccf56d7e0246d3baf3fdbc11138b32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:19:19.932700 19869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: acquiring CA information for follower catalog manager: success
I20260924 20:19:19.933888 19869 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2927826fb15b43de8648461cd321add9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:19:20.181571 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:20.177393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.29.83.189:41567)
0924 20:19:20.177653 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:20.177657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:20.177673 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:19:20.177724 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:20.177727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:20.177880 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:20.177983 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:20.178309 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.178969 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.179907 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.180148 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.180238 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:20.180575 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:20.181078 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:20.181177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:20.181348 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 20:19:20.181428 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260924 20:19:20.190073 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:20.185022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.29.83.188:46363)
0924 20:19:20.185273 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:20.185277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:20.185645 (+ 368us) server_negotiation.cc:407] Connection header received
0924 20:19:20.186047 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:20.186049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:20.186171 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:20.186310 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:20.186686 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.187207 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.188162 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.188527 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.188646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:20.189007 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:20.189475 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:20.189577 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:20.189735 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 20:19:20.189878 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260924 20:19:20.198150 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:20.193938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.29.83.188:46363)
0924 20:19:20.194094 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:20.194098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:20.194316 (+ 218us) server_negotiation.cc:407] Connection header received
0924 20:19:20.194485 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:20.194489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:20.194539 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:20.194616 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:20.194919 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.195534 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.196873 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.197211 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.197311 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:20.197630 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:20.197718 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:20.197822 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:20.197943 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 20:19:20.198008 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:19:20.198163 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:20.193809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.29.83.189:41567)
0924 20:19:20.193925 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:20.193930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:20.194331 (+ 401us) server_negotiation.cc:407] Connection header received
0924 20:19:20.194570 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:20.194576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:20.194647 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:20.194746 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:20.194990 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.195835 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.196941 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.197281 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.197384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:20.197675 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:20.197791 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:20.197854 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:20.198003 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 20:19:20.198056 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 20:19:20.199055 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:20.193714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.29.83.190:45249)
0924 20:19:20.194068 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:20.194072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:20.194405 (+ 333us) server_negotiation.cc:407] Connection header received
0924 20:19:20.194577 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:20.194583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:20.194759 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:20.194885 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:20.195101 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.195905 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.197123 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:20.197491 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:20.197606 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:20.198072 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:20.198465 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:20.198582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:20.198847 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 20:19:20.198910 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":84,"threads_started":1}
I20260924 20:19:20.202855 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19600
I20260924 20:19:20.210165 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19675
I20260924 20:19:20.216495 30030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrGCYXn/build/debug/bin/kudu with pid 19749
2026-09-24T20:19:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3214 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6502 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375591 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS